Jul 09 18:02:57.780553 ip-10-0-141-82 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:02:58.239271 ip-10-0-141-82 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:02:58.239271 ip-10-0-141-82 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:02:58.239271 ip-10-0-141-82 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:02:58.239271 ip-10-0-141-82 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:02:58.239271 ip-10-0-141-82 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:02:58.240613 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.240521    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:02:58.244749 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244731    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.244749 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244748    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.244749 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244752    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244755    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244759    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244761    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244764    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244767    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244769    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244772    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244775    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244777    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244780    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244783    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244786    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244794    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244797    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244799    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244803    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244806    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244809    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244811    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.244849 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244814    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244817    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244820    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244822    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244825    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244828    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244830    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244833    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244836    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244838    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244840    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244843    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244846    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244850    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244853    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244857    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244863    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244866    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244870    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.245375 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244873    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244875    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244878    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244881    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244884    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244887    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244890    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244893    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244896    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244899    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244902    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244905    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244908    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244911    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244913    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244916    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244918    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244921    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244924    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244929    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.245818 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244932    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244935    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244939    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244942    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244945    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244948    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244951    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244956    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244958    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244961    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244963    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244966    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244968    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244971    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244974    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244977    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244979    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244982    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244984    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.246323 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244987    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244989    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244991    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244994    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244996    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.244999    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245002    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245005    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245008    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245010    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245012    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245015    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245017    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245020    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245023    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245025    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245028    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245030    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245034    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245815    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.246775 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245823    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245827    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245830    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245834    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245839    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245844    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245847    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245850    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245853    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245856    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245859    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245862    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245866    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245869    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245873    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245876    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245879    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245881    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245884    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.247305 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245887    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245890    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245893    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245895    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245898    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245901    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245904    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245907    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245909    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245911    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245917    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245920    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245923    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245925    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245928    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245931    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245934    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245936    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245939    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.247784 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245942    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245944    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245947    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245950    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245952    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245955    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245957    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245960    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245962    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245965    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245968    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245971    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245973    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245975    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245978    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245980    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245983    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245985    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245988    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245991    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.248332 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245994    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.245998    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246002    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246005    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246008    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246011    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246014    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246017    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246020    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246022    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246025    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246027    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246030    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246032    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246035    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246038    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246041    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246043    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246046    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.248819 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246048    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246051    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246053    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246056    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246058    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246060    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246063    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246066    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246068    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246070    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246073    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246075    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246077    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246080    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246082    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246085    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246088    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246091    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246093    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246096    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.249290 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246526    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246532    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246535    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246538    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246540    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246543    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246545    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246548    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246551    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246554    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246557    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246559    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246562    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246564    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246567    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246569    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246572    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246574    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246577    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.249773 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246579    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246582    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246584    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246587    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246589    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246592    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246594    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246597    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246600    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246603    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246605    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246609    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246611    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246614    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246616    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246619    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246621    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246624    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246627    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.250246 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246629    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246632    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246635    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246638    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246640    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246643    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246645    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246648    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246651    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246654    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246656    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246658    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246661    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246663    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246668    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246671    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246673    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246675    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246678    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246682    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.250722 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246684    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246687    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246690    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246692    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246695    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246698    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246701    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246704    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246706    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246709    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246711    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246714    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246717    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246720    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246722    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246728    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246731    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246734    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246736    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246739    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.251202 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246741    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246744    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246747    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246749    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246752    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246754    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246757    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246759    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246762    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246766    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246769    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246773    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246775    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246778    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246781    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246786    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246790    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246793    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246796    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.251709 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246798    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.246801    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246876    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246883    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246892    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246899    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246905    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246910    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246920    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246924    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246928    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246931    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246935    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246938    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246941    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246944    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246947    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246951    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246954    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246957    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246961    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246964    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246967    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246970    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:02:58.252186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246973    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246977    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246980    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246984    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246987    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246990    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246993    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.246997    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247000    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247003    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247008    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247010    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247014    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247016    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247020    2565 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247022    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247030    2565 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247033    2565 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247036    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247040    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247043    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247047    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247050    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247053    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247056    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:02:58.252776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247059    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247062    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247065    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247068    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247071    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247074    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247077    2565 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247081    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247084    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247087    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247090    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247093    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247098    2565 flags.go:64] FLAG: --help="false"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247100    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247104    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247107    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247110    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247113    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247116    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247119    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247122    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247125    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247128    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:02:58.253382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247132    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247136    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247139    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247142    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247145    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247149    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247151    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247154    2565 flags.go:64] FLAG: --lock-file=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247157    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247160    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247163    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247168    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247171    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247174    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247176    2565 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247179    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247183    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247185    2565 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247188    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247193    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247196    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247201    2565 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247205    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247208    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247211    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:02:58.253957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247214    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247217    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247220    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247223    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247242    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247245    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247248    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247253    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247256    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247262    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247266    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247269    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247276    2565 flags.go:64] FLAG: --port="10250"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247279    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247282    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aade291f45cf986d"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247286    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247289    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247292    2565 flags.go:64] FLAG: --register-node="true"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247295    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247299    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247302    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247305    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247309    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247312    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247316    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:02:58.254693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247319    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247322    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247325    2565 flags.go:64] FLAG: --runonce="false"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247329    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247333    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247336    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247339    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247341    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247344    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247348    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247351    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247354    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247357    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247359    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247362    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247367    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247370    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247373    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247384    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247387    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247396    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247400    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247403    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247406    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247409    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:02:58.255333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247412    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247415    2565 flags.go:64] FLAG: --v="2"
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247419    2565 flags.go:64] FLAG: --version="false"
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247423    2565 flags.go:64] FLAG: --vmodule=""
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247426    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.247430    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247599    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247603    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247607    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247611    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247617    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247620    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247623    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247625    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247630    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247633    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247636    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247639    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247641    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247644    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.256097 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247647    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247649    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247653    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247656    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247658    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247661    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247664    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247666    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247669    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247672    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247675    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247678    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247680    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247683    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247685    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247688    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247690    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247693    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247695    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247698    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.256616 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247700    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247703    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247707    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247709    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247712    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247714    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247716    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247719    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247722    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247724    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247727    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247730    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247732    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247735    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247738    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247742    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247746    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247748    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247751    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247754    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.257110 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247756    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247760    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247763    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247765    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247768    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247770    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247773    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247775    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247778    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247781    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247783    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247786    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247789    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247792    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247804    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247807    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247809    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247812    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247815    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247818    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.257654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247821    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247823    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247825    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247828    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247831    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247834    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247837    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247839    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247842    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247844    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247847    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247849    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247852    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247855    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247859    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247861    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247864    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247866    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247869    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247871    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.258141 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247874    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247876    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247880    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247883    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.247886    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.248617    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.255956    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.255972    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.255990    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.256000    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256056    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256061    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256064    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256068    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.258638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256071    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256074    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256076    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256079    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256082    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256085    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256088    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256092    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256095    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256099    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256102    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256105    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256108    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256111    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256114    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256117    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256119    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256122    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256125    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.258995 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256128    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256132    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256136    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256139    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256142    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256144    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256149    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256152    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256154    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256157    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256160    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256163    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256165    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256168    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256170    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256173    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256175    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256178    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256180    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256183    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.259479 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256186    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256188    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256191    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256193    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256196    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256198    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256201    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256203    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256206    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256208    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256211    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256213    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256216    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256218    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256220    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256223    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256226    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256243    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256248    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256251    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.259971 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256254    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256257    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256259    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256262    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256264    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256267    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256270    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256272    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256275    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256277    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256280    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256282    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256285    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256287    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256290    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256292    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256295    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256297    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256300    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256302    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.260471 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256305    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256307    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256309    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256312    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256314    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256317    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256319    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256322    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256325    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256329    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256332    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256336    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256339    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256343    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256346    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.260954 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256348    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.256353    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256467    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256472    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256475    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256477    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256480    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256483    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256485    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256488    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256491    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256493    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256496    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256500    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:02:58.261492 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256504    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256507    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256510    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256512    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256515    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256518    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256521    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256524    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256526    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256529    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256532    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256534    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256537    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256540    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256542    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256545    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256548    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256550    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256553    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256555    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:02:58.261857 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256558    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256560    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256563    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256565    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256568    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256570    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256573    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256575    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256577    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256580    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256582    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256585    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256587    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256590    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256592    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256595    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256597    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256599    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256602    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256604    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:02:58.262559 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256607    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256609    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256612    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256614    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256617    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256620    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256622    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256625    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256628    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256631    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256633    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256637    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256640    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256643    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256645    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256648    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256651    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256653    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256656    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:02:58.263336 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256659    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256661    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256663    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256666    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256669    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256671    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256674    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256676    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256679    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256681    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256684    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256686    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256689    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256691    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256693    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256696    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256699    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256702    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256704    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:02:58.263985 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256708    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256711    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256714    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256717    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256720    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256722    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256725    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256727    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:58.256729    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.256734    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.256863    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.259299    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.261133    2565 server.go:1019] "Starting client certificate rotation"
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.263739    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:02:58.264846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.264332    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:02:58.268173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.268152    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:02:58.272476 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.272460    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:02:58.283282 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.283263    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:02:58.285766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.285748    2565 log.go:25] "Validated CRI v1 image API"
Jul 09 18:02:58.289007 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.288990    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:02:58.291724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.291694    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bffe7b7-f590-425d-a622-6e3a137b66b9:/dev/nvme0n1p3 a4f7cbe3-03b9-4be7-8e10-2b5535c42dff:/dev/nvme0n1p4]
Jul 09 18:02:58.291823 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.291723    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:02:58.299109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.298990    2565 manager.go:216] Machine: {Timestamp:2026-07-09 18:02:58.296526506 +0000 UTC m=+0.383775801 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106622 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2338b280cddee0b739d3b5d923f16e SystemUUID:ec2338b2-80cd-dee0-b739-d3b5d923f16e BootID:e4f6563b-3b8d-43c1-b56d-2a7edcfd9adc Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:9a:06:5f:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:9a:06:5f:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:e7:4f:a3:f6:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:02:58.299109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.299096    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:02:58.299227 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.299190    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:02:58.302676 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.302636    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:02:58.302806 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.302673    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:02:58.304279 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.304268    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:02:58.304317 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.304282    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:02:58.304317 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.304289    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:02:58.304640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.304624    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:02:58.305122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.305110    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:02:58.306998 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.306987    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:02:58.311882 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.311869    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:02:58.312029 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312020    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:02:58.312062 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312057    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:02:58.312089 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312067    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:02:58.312089 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312077    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:02:58.312089 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312087    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:02:58.312180 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312096    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:02:58.312624 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312598    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:02:58.312624 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.312622    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:02:58.316434 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.316409    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:02:58.319094 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319075    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:02:58.319192 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319115    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:02:58.319340 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319327    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:02:58.319409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319397    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:02:58.319460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319412    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:02:58.319460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319422    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:02:58.319460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319434    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:02:58.319460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319442    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:02:58.319836 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319824    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:02:58.319898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319837    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:02:58.319898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319854    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:02:58.319898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319863    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:02:58.319898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319876    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:02:58.319898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.319887    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:02:58.320715 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.320703    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:02:58.320715 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.320717    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:02:58.324324 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.324308    2565 server.go:1268] "Started kubelet"
Jul 09 18:02:58.324426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.324399    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:02:58.324569 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.324540    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:02:58.324667 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.324654    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:02:58.325426 ip-10-0-141-82 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:02:58.325950 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.325549    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:02:58.325950 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.325899    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:02:58.325950 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.325901    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:02:58.326107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.326025    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:02:58.326107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.326068    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:02:58.331364 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.330204    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-82.ec2.internal.18c0b11a273cdd34  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-82.ec2.internal,UID:ip-10-0-141-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-82.ec2.internal,},FirstTimestamp:2026-07-09 18:02:58.324282676 +0000 UTC m=+0.411531970,LastTimestamp:2026-07-09 18:02:58.324282676 +0000 UTC m=+0.411531970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-82.ec2.internal,}"
Jul 09 18:02:58.331582 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.331566    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:02:58.335772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.335753    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:02:58.335772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.335759    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:02:58.335918 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.335870    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.336594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336559    2565 factory.go:55] Registering systemd factory
Jul 09 18:02:58.336594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336572    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:02:58.336594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336578    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:02:58.336594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336583    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:02:58.336594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336598    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:02:58.336811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336666    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:02:58.336811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.336675    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:02:58.337138 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337117    2565 factory.go:153] Registering CRI-O factory
Jul 09 18:02:58.337138 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337132    2565 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:02:58.337263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337208    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:02:58.337263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337254    2565 factory.go:103] Registering Raw factory
Jul 09 18:02:58.337375 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337270    2565 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:02:58.337839 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.337815    2565 manager.go:349] Starting recovery of all containers
Jul 09 18:02:58.338337 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.338310    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:02:58.338557 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.338527    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:02:58.352003 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.351981    2565 manager.go:354] Recovery completed
Jul 09 18:02:58.353947 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.353919    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xqsjp"
Jul 09 18:02:58.356531 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.356518    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.358966 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.358950    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.359056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.358982    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.359056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.358995    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.359640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.359623    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:02:58.359640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.359639    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:02:58.359750 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.359648    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.359750 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.359659    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:02:58.360818 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.360801    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.360877 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.360830    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.360877 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.360839    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.361906 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.361893    2565 policy_none.go:49] "None policy: Start"
Jul 09 18:02:58.361943 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.361911    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:02:58.361943 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.361926    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:02:58.362128 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.362114    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xqsjp"
Jul 09 18:02:58.364011 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.363998    2565 policy_none.go:47] "Start"
Jul 09 18:02:58.401969 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.401941    2565 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:02:58.402106 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.401980    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:02:58.402106 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.401993    2565 server.go:87] "Starting device plugin registration server"
Jul 09 18:02:58.402389 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.402288    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:02:58.402389 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.402299    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:02:58.403539 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.402470    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:02:58.403539 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.402528    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:02:58.403539 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.402534    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:02:58.403539 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.403171    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:02:58.403539 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.403223    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.474463 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.474284    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:02:58.475639 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.475622    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:02:58.475694 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.475660    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:02:58.475694 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.475682    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:02:58.475770 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.475714    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:02:58.478163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.478139    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:02:58.503147 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.503074    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.504064 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.504046    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.504169 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.504073    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.504169 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.504085    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.504169 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.504106    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.513884 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.513867    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.513948 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.513892    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-82.ec2.internal\": node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.544192 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.544165    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.576012 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.575948    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"]
Jul 09 18:02:58.576103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.576065    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.577084 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.577061    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.577151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.577090    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.577151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.577107    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.579570 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.579557    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.579671 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.579659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.579704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.579700    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.580358 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580344    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.580358 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580353    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.580460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580368    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.580460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580374    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.580460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580378    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.580460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.580383    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.582451 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.582435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.582491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.582467    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:02:58.583154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.583137    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:02:58.583217 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.583169    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:02:58.583217 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.583183    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:02:58.598910 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.598886    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.ec2.internal\" not found" node="ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.602334 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.602314    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.ec2.internal\" not found" node="ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.638143 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.638107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.638143 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.638147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.638382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.638173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52024e858ee987f467ceba75222ec174-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"52024e858ee987f467ceba75222ec174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.645259 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.645215    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.738808 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.738964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.738964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52024e858ee987f467ceba75222ec174-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"52024e858ee987f467ceba75222ec174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.738964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.738964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b88f81944f9da8e5948e6d8adf8450a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"3b88f81944f9da8e5948e6d8adf8450a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.738964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.738894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52024e858ee987f467ceba75222ec174-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"52024e858ee987f467ceba75222ec174\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.745888 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.745870    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.846749 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.846713    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:58.904349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.904300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.905487 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:58.905470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:58.947695 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:58.947663    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.048405 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:59.048328    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.149474 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:59.149432    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.250618 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:59.250575    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.260454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.260426    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:02:59.260629 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.260610    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:02:59.351058 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:59.350991    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.366936 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.366884    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:57:59 +0000 UTC" deadline="2027-12-03 16:13:21.60695825 +0000 UTC"
Jul 09 18:02:59.366936 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.366925    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12286h10m22.240036668s"
Jul 09 18:02:59.451302 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:02:59.451268    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found"
Jul 09 18:02:59.534370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.534346    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:02:59.536285 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.536266    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:59.551065 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.551046    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:02:59.552654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.552635    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"
Jul 09 18:02:59.562812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.562784    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:02:59.624035 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.624009    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:02:59.667630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.667601    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:02:59.742751 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:02:59.742706    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b88f81944f9da8e5948e6d8adf8450a.slice/crio-779a6285679be7391605b133dde6b06c9f610f87ae8f444f862427ac2e951c53 WatchSource:0}: Error finding container 779a6285679be7391605b133dde6b06c9f610f87ae8f444f862427ac2e951c53: Status 404 returned error can't find the container with id 779a6285679be7391605b133dde6b06c9f610f87ae8f444f862427ac2e951c53
Jul 09 18:02:59.747226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:02:59.747203    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:03:00.256216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.256179    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:00.315421 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.315386    2565 apiserver.go:52] "Watching apiserver"
Jul 09 18:03:00.322187 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.322163    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:03:00.322605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.322574    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-h7vxc","kube-system/global-pull-secret-syncer-kmbmg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l","openshift-dns/node-resolver-82jsv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal","openshift-multus/multus-zwftt","openshift-ovn-kubernetes/ovnkube-node-zqxhn","kube-system/konnectivity-agent-87jjw","kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qjvln","openshift-image-registry/node-ca-jdz76","openshift-multus/multus-additional-cni-plugins-cbckn","openshift-multus/network-metrics-daemon-hjjkk","openshift-network-diagnostics/network-check-target-njgpn"]
Jul 09 18:03:00.326622 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.326606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.328942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.328924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.329546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329513    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:03:00.329668 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329583    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:03:00.329668 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:03:00.329668 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329610    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.329841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329587    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.329841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329701    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j7xkq\""
Jul 09 18:03:00.329841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.329724    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:03:00.331262 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.331235    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.331418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.331394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.331662 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.331557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:03:00.331856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.331839    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w47d9\""
Jul 09 18:03:00.332037 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.332020    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.333562 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.333541    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.333931 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.333912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.334161 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.334136    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:03:00.334161 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.334138    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9m72l\""
Jul 09 18:03:00.334337 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.334247    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.336022 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.336006    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:00.336182 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.336140    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.336316 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.336298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:00.336395 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.336372    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:00.336526 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.336510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-skzfs\""
Jul 09 18:03:00.336681 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.336662    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.338964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.338946    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.341456 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.341440    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.341668 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.341648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:03:00.341778 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.341669    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:03:00.341885 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.341863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.342554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.342354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4kzsw\""
Jul 09 18:03:00.342554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.342354    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.344423 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.343961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:03:00.344423 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.344212    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:03:00.344423 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.344391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7n5kq\""
Jul 09 18:03:00.345980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.345960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.346079 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.346021    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:00.348171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348151    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:00.348493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-slash\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-netns\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhtl\" (UniqueName: \"kubernetes.io/projected/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-kube-api-access-rvhtl\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnv6\" (UniqueName: \"kubernetes.io/projected/44981ba1-e2ac-4bca-83c9-6988a5237880-kube-api-access-9wnv6\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.348585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-registration-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-netns\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-hostroot\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.348656    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-ovn\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-log-socket\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.348794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-env-overrides\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44981ba1-e2ac-4bca-83c9-6988a5237880-host\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-socket-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-socket-dir-parent\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-systemd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovn-node-metrics-cert\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-script-lib\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-bin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.348988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-multus\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-kubelet\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-var-lib-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-node-log\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-config\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-device-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-system-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-sys-fs\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lq2\" (UniqueName: \"kubernetes.io/projected/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kube-api-access-q6lq2\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430784f-f90b-4d10-9461-d80c0d848b40-tmp-dir\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-cnibin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-cni-binary-copy\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-daemon-config\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-multus-certs\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-os-release\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-k8s-cni-cncf-io\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-kubelet\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-bin\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.349631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-netd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6430784f-f90b-4d10-9461-d80c0d848b40-hosts-file\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-etc-kubernetes\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmnd\" (UniqueName: \"kubernetes.io/projected/6b0a56a1-54a9-4456-837a-a7ae772a9716-kube-api-access-xgmnd\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-systemd-units\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-etc-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44981ba1-e2ac-4bca-83c9-6988a5237880-serviceca\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5k2\" (UniqueName: \"kubernetes.io/projected/6430784f-f90b-4d10-9461-d80c0d848b40-kube-api-access-kr5k2\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.350162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.349823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-conf-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.351077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.351059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.353657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.353547    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.353657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.353579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.353657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.353612    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dxt28\""
Jul 09 18:03:00.353657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.353626    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.356167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.356140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.356247 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.356174    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:00.356247 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.356219    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:00.356511 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.356451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:03:00.356662 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.356646    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vt9gw\""
Jul 09 18:03:00.358748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.358732    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:03:00.358826 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.358770    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mc6dr\""
Jul 09 18:03:00.359104 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.359090    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:03:00.385060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.385040    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvrhw"
Jul 09 18:03:00.395318 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.395300    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvrhw"
Jul 09 18:03:00.438159 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.438138    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:03:00.450386 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-systemd-units\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44981ba1-e2ac-4bca-83c9-6988a5237880-serviceca\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.450561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-systemd-units\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-conf-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.450669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-system-cni-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.450669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmnd\" (UniqueName: \"kubernetes.io/projected/6b0a56a1-54a9-4456-837a-a7ae772a9716-kube-api-access-xgmnd\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.450669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-conf-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.450669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-slash\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhtl\" (UniqueName: \"kubernetes.io/projected/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-kube-api-access-rvhtl\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-systemd\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-ovn\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-slash\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-ovn\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-socket-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.450908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-socket-dir-parent\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.450942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-socket-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-os-release\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44981ba1-e2ac-4bca-83c9-6988a5237880-serviceca\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-socket-dir-parent\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-systemd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovn-node-metrics-cert\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-systemd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-multus\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-kubernetes\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-lib-modules\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-multus\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-node-log\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-config\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:00.451417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-node-log\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-cni-binary-copy\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-daemon-config\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-multus-certs\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-kubelet-config\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysconfig\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-os-release\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451601    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-os-release\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-k8s-cni-cncf-io\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-dbus\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-multus-certs\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmj5p\" (UniqueName: \"kubernetes.io/projected/0fdc1f91-8a06-4af0-95df-82e847e1e57d-kube-api-access-nmj5p\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-host-slash\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-netd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-conf\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-k8s-cni-cncf-io\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-run-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.451997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-config\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-host\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-daemon-config\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-netd\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0a56a1-54a9-4456-837a-a7ae772a9716-cni-binary-copy\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-etc-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5k2\" (UniqueName: \"kubernetes.io/projected/6430784f-f90b-4d10-9461-d80c0d848b40-kube-api-access-kr5k2\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-etc-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-modprobe-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-var-lib-kubelet\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-iptables-alerter-script\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.452972 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-etc-kubernetes\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-netns\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-etc-kubernetes\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-netns\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnv6\" (UniqueName: \"kubernetes.io/projected/44981ba1-e2ac-4bca-83c9-6988a5237880-kube-api-access-9wnv6\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-registration-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-netns\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-hostroot\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-hostroot\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-registration-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-run-netns\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-agent-certs\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-log-socket\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-env-overrides\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44981ba1-e2ac-4bca-83c9-6988a5237880-host\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452802    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-log-socket\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbmp\" (UniqueName: \"kubernetes.io/projected/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-kube-api-access-hqbmp\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.453783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-run\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44981ba1-e2ac-4bca-83c9-6988a5237880-host\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-sys\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-tmp\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.452997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-script-lib\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-bin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-kubelet\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xsn\" (UniqueName: \"kubernetes.io/projected/4df6f073-d9d4-4339-8fa3-8605f157a4e2-kube-api-access-q2xsn\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-var-lib-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-kubelet\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-host-var-lib-cni-bin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-var-lib-openvswitch\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-multus-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.454661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-device-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-system-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-device-dir\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqpc2\" (UniqueName: \"kubernetes.io/projected/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-kube-api-access-mqpc2\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-system-cni-dir\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cnibin\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-sys-fs\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lq2\" (UniqueName: \"kubernetes.io/projected/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kube-api-access-q6lq2\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430784f-f90b-4d10-9461-d80c0d848b40-tmp-dir\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd0ef01d-cc84-452f-8e57-1d72aff552fc-sys-fs\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-cnibin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-tuned\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-konnectivity-ca\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-env-overrides\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-kubelet\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.455513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovnkube-script-lib\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-bin\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453807    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0a56a1-54a9-4456-837a-a7ae772a9716-cnibin\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6430784f-f90b-4d10-9461-d80c0d848b40-hosts-file\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-kubelet\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-host-cni-bin\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6430784f-f90b-4d10-9461-d80c0d848b40-hosts-file\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.453926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430784f-f90b-4d10-9461-d80c0d848b40-tmp-dir\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.456172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.456091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-ovn-node-metrics-cert\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.457485 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.457467    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:00.457542 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.457490    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:00.457542 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.457500    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:00.457608 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.457560    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:00.957544981 +0000 UTC m=+3.044794285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:00.459742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.459717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhtl\" (UniqueName: \"kubernetes.io/projected/dd9e7bc5-49c5-4535-b31c-b9f2b594e74f-kube-api-access-rvhtl\") pod \"ovnkube-node-zqxhn\" (UID: \"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.459828 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.459818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmnd\" (UniqueName: \"kubernetes.io/projected/6b0a56a1-54a9-4456-837a-a7ae772a9716-kube-api-access-xgmnd\") pod \"multus-zwftt\" (UID: \"6b0a56a1-54a9-4456-837a-a7ae772a9716\") " pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.462266 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.462238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnv6\" (UniqueName: \"kubernetes.io/projected/44981ba1-e2ac-4bca-83c9-6988a5237880-kube-api-access-9wnv6\") pod \"node-ca-jdz76\" (UID: \"44981ba1-e2ac-4bca-83c9-6988a5237880\") " pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.462851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.462827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5k2\" (UniqueName: \"kubernetes.io/projected/6430784f-f90b-4d10-9461-d80c0d848b40-kube-api-access-kr5k2\") pod \"node-resolver-82jsv\" (UID: \"6430784f-f90b-4d10-9461-d80c0d848b40\") " pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.467484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.467461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lq2\" (UniqueName: \"kubernetes.io/projected/dd0ef01d-cc84-452f-8e57-1d72aff552fc-kube-api-access-q6lq2\") pod \"aws-ebs-csi-driver-node-d4n2l\" (UID: \"dd0ef01d-cc84-452f-8e57-1d72aff552fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.484513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.484472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"3b88f81944f9da8e5948e6d8adf8450a","Type":"ContainerStarted","Data":"779a6285679be7391605b133dde6b06c9f610f87ae8f444f862427ac2e951c53"}
Jul 09 18:03:00.554902 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-iptables-alerter-script\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.554902 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-agent-certs\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.554902 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbmp\" (UniqueName: \"kubernetes.io/projected/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-kube-api-access-hqbmp\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-run\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-sys\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-tmp\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.554985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xsn\" (UniqueName: \"kubernetes.io/projected/4df6f073-d9d4-4339-8fa3-8605f157a4e2-kube-api-access-q2xsn\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-run\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqpc2\" (UniqueName: \"kubernetes.io/projected/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-kube-api-access-mqpc2\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.555167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-sys\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cnibin\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-tuned\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cnibin\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-konnectivity-ca\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-system-cni-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-systemd\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-os-release\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.555571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-kubernetes\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-iptables-alerter-script\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-lib-modules\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-kubelet-config\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.555683    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysconfig\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-system-cni-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-dbus\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmj5p\" (UniqueName: \"kubernetes.io/projected/0fdc1f91-8a06-4af0-95df-82e847e1e57d-kube-api-access-nmj5p\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-host-slash\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.555794    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:01.055776959 +0000 UTC m=+3.143026262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-conf\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-lib-modules\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-host-slash\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-host\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-host\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.555958    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:00.556173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-systemd\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.555979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-os-release\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.556011    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:01.055994057 +0000 UTC m=+3.143243341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-kubelet-config\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysconfig\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b38b34-371d-43da-9ed4-6da5eb6ee973-dbus\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-sysctl-conf\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-kubernetes\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-modprobe-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-var-lib-kubelet\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-var-lib-kubelet\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-modprobe-d\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4df6f073-d9d4-4339-8fa3-8605f157a4e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557916 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4df6f073-d9d4-4339-8fa3-8605f157a4e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.557916 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.556848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-konnectivity-ca\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.558017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.557996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-tmp\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.558170 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.558138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fdc1f91-8a06-4af0-95df-82e847e1e57d-etc-tuned\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.558286 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.558264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3022f01d-3e8d-4f25-964b-f0cfbdd07da6-agent-certs\") pod \"konnectivity-agent-87jjw\" (UID: \"3022f01d-3e8d-4f25-964b-f0cfbdd07da6\") " pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.564164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.563728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqpc2\" (UniqueName: \"kubernetes.io/projected/65f8c8c3-8000-4495-bebc-921e1f1a5a9b-kube-api-access-mqpc2\") pod \"iptables-alerter-h7vxc\" (UID: \"65f8c8c3-8000-4495-bebc-921e1f1a5a9b\") " pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.564164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.563838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xsn\" (UniqueName: \"kubernetes.io/projected/4df6f073-d9d4-4339-8fa3-8605f157a4e2-kube-api-access-q2xsn\") pod \"multus-additional-cni-plugins-cbckn\" (UID: \"4df6f073-d9d4-4339-8fa3-8605f157a4e2\") " pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.564444 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.564424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbmp\" (UniqueName: \"kubernetes.io/projected/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-kube-api-access-hqbmp\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:00.565528 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.565508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmj5p\" (UniqueName: \"kubernetes.io/projected/0fdc1f91-8a06-4af0-95df-82e847e1e57d-kube-api-access-nmj5p\") pod \"tuned-qjvln\" (UID: \"0fdc1f91-8a06-4af0-95df-82e847e1e57d\") " pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.566516 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.566492    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52024e858ee987f467ceba75222ec174.slice/crio-c1f392bec5181d7c1dcd2ea2bab5381279420fd92a8dc47415cdc97a2f090df1 WatchSource:0}: Error finding container c1f392bec5181d7c1dcd2ea2bab5381279420fd92a8dc47415cdc97a2f090df1: Status 404 returned error can't find the container with id c1f392bec5181d7c1dcd2ea2bab5381279420fd92a8dc47415cdc97a2f090df1
Jul 09 18:03:00.637441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.637402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:00.644007 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.643980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdz76"
Jul 09 18:03:00.647944 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.647912    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9e7bc5_49c5_4535_b31c_b9f2b594e74f.slice/crio-2e95a3e24a13278bdab73cbe64a4798a976df0aa09bd7f0e274e542fcf55e2b3 WatchSource:0}: Error finding container 2e95a3e24a13278bdab73cbe64a4798a976df0aa09bd7f0e274e542fcf55e2b3: Status 404 returned error can't find the container with id 2e95a3e24a13278bdab73cbe64a4798a976df0aa09bd7f0e274e542fcf55e2b3
Jul 09 18:03:00.652386 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.652361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l"
Jul 09 18:03:00.652788 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.652760    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44981ba1_e2ac_4bca_83c9_6988a5237880.slice/crio-60e081ccc5435ac695435c1135a518424ebb49ce698adcda5b5722f8787f574e WatchSource:0}: Error finding container 60e081ccc5435ac695435c1135a518424ebb49ce698adcda5b5722f8787f574e: Status 404 returned error can't find the container with id 60e081ccc5435ac695435c1135a518424ebb49ce698adcda5b5722f8787f574e
Jul 09 18:03:00.657554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.657531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82jsv"
Jul 09 18:03:00.662331 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.662292    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0ef01d_cc84_452f_8e57_1d72aff552fc.slice/crio-5384b95620accf3f8c443c3c9aa6ca6fffc841bdcd19424982b2bfda94c15658 WatchSource:0}: Error finding container 5384b95620accf3f8c443c3c9aa6ca6fffc841bdcd19424982b2bfda94c15658: Status 404 returned error can't find the container with id 5384b95620accf3f8c443c3c9aa6ca6fffc841bdcd19424982b2bfda94c15658
Jul 09 18:03:00.663514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.663493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwftt"
Jul 09 18:03:00.668670 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.668642    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6430784f_f90b_4d10_9461_d80c0d848b40.slice/crio-3fdbd005ceed0ea6efa0800b6704a32e00d27c969e920d97219ed1d444404476 WatchSource:0}: Error finding container 3fdbd005ceed0ea6efa0800b6704a32e00d27c969e920d97219ed1d444404476: Status 404 returned error can't find the container with id 3fdbd005ceed0ea6efa0800b6704a32e00d27c969e920d97219ed1d444404476
Jul 09 18:03:00.669429 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.669407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:00.675749 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.675727    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjvln"
Jul 09 18:03:00.676194 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.676173    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0a56a1_54a9_4456_837a_a7ae772a9716.slice/crio-019cb1bd83502895b71cbf06f6f33e79a6ba28c0e2c080fd9afd6a1a84e6ca29 WatchSource:0}: Error finding container 019cb1bd83502895b71cbf06f6f33e79a6ba28c0e2c080fd9afd6a1a84e6ca29: Status 404 returned error can't find the container with id 019cb1bd83502895b71cbf06f6f33e79a6ba28c0e2c080fd9afd6a1a84e6ca29
Jul 09 18:03:00.679736 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.679688    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3022f01d_3e8d_4f25_964b_f0cfbdd07da6.slice/crio-0ef063d87508228bd0075a6a9d2da6579661eb87edcca570e6fe4b44cddca728 WatchSource:0}: Error finding container 0ef063d87508228bd0075a6a9d2da6579661eb87edcca570e6fe4b44cddca728: Status 404 returned error can't find the container with id 0ef063d87508228bd0075a6a9d2da6579661eb87edcca570e6fe4b44cddca728
Jul 09 18:03:00.682074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.682054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7vxc"
Jul 09 18:03:00.685502 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.685479    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdc1f91_8a06_4af0_95df_82e847e1e57d.slice/crio-3115d2d7325c3e741604fc727e0bd5922f2c67856a0122392fa04ce52fa52ae2 WatchSource:0}: Error finding container 3115d2d7325c3e741604fc727e0bd5922f2c67856a0122392fa04ce52fa52ae2: Status 404 returned error can't find the container with id 3115d2d7325c3e741604fc727e0bd5922f2c67856a0122392fa04ce52fa52ae2
Jul 09 18:03:00.686168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.686002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbckn"
Jul 09 18:03:00.693654 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.693634    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f8c8c3_8000_4495_bebc_921e1f1a5a9b.slice/crio-e0f16c89ad8dd274076087df04c76a99e798f9f8cb3952f9c91ea2ec603fda29 WatchSource:0}: Error finding container e0f16c89ad8dd274076087df04c76a99e798f9f8cb3952f9c91ea2ec603fda29: Status 404 returned error can't find the container with id e0f16c89ad8dd274076087df04c76a99e798f9f8cb3952f9c91ea2ec603fda29
Jul 09 18:03:00.695560 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:00.695540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df6f073_d9d4_4339_8fa3_8605f157a4e2.slice/crio-b7caece55a5423c49eb9f8b8b6ebf0df5cc64486063141da14c95e144023f0ac WatchSource:0}: Error finding container b7caece55a5423c49eb9f8b8b6ebf0df5cc64486063141da14c95e144023f0ac: Status 404 returned error can't find the container with id b7caece55a5423c49eb9f8b8b6ebf0df5cc64486063141da14c95e144023f0ac
Jul 09 18:03:00.960748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:00.960656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:00.960902 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.960795    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:00.960902 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.960812    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:00.960902 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.960822    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:00.960902 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:00.960880    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:01.960862072 +0000 UTC m=+4.048111357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:01.061301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.061261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:01.061507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.061348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:01.061507 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.061472    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:01.061507 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.061496    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:01.061645 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.061548    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:02.061529494 +0000 UTC m=+4.148778800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:01.061645 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.061564    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:02.061557526 +0000 UTC m=+4.148806809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:01.398530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.398396    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:01 +0000 UTC" deadline="2027-12-22 19:38:09.133422288 +0000 UTC"
Jul 09 18:03:01.398530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.398475    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12745h35m7.734952579s"
Jul 09 18:03:01.491016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.490293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdz76" event={"ID":"44981ba1-e2ac-4bca-83c9-6988a5237880","Type":"ContainerStarted","Data":"60e081ccc5435ac695435c1135a518424ebb49ce698adcda5b5722f8787f574e"}
Jul 09 18:03:01.492698 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.491765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" event={"ID":"52024e858ee987f467ceba75222ec174","Type":"ContainerStarted","Data":"c1f392bec5181d7c1dcd2ea2bab5381279420fd92a8dc47415cdc97a2f090df1"}
Jul 09 18:03:01.493634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.493580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7vxc" event={"ID":"65f8c8c3-8000-4495-bebc-921e1f1a5a9b","Type":"ContainerStarted","Data":"e0f16c89ad8dd274076087df04c76a99e798f9f8cb3952f9c91ea2ec603fda29"}
Jul 09 18:03:01.495782 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.495524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l" event={"ID":"dd0ef01d-cc84-452f-8e57-1d72aff552fc","Type":"ContainerStarted","Data":"5384b95620accf3f8c443c3c9aa6ca6fffc841bdcd19424982b2bfda94c15658"}
Jul 09 18:03:01.497404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.497375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"2e95a3e24a13278bdab73cbe64a4798a976df0aa09bd7f0e274e542fcf55e2b3"}
Jul 09 18:03:01.504664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.504632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerStarted","Data":"b7caece55a5423c49eb9f8b8b6ebf0df5cc64486063141da14c95e144023f0ac"}
Jul 09 18:03:01.508557 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.507886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjvln" event={"ID":"0fdc1f91-8a06-4af0-95df-82e847e1e57d","Type":"ContainerStarted","Data":"3115d2d7325c3e741604fc727e0bd5922f2c67856a0122392fa04ce52fa52ae2"}
Jul 09 18:03:01.511868 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.511790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87jjw" event={"ID":"3022f01d-3e8d-4f25-964b-f0cfbdd07da6","Type":"ContainerStarted","Data":"0ef063d87508228bd0075a6a9d2da6579661eb87edcca570e6fe4b44cddca728"}
Jul 09 18:03:01.514897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.514864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwftt" event={"ID":"6b0a56a1-54a9-4456-837a-a7ae772a9716","Type":"ContainerStarted","Data":"019cb1bd83502895b71cbf06f6f33e79a6ba28c0e2c080fd9afd6a1a84e6ca29"}
Jul 09 18:03:01.517146 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.517091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82jsv" event={"ID":"6430784f-f90b-4d10-9461-d80c0d848b40","Type":"ContainerStarted","Data":"3fdbd005ceed0ea6efa0800b6704a32e00d27c969e920d97219ed1d444404476"}
Jul 09 18:03:01.969922 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:01.969879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:01.970139 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.970086    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:01.970139 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.970109    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:01.970139 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.970124    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:01.970353 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:01.970191    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:03.970172871 +0000 UTC m=+6.057422172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:02.070742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.070700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:02.070937 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.070779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:02.070937 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.070864    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:02.070937 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.070921    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:02.071104 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.070951    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:04.070927723 +0000 UTC m=+6.158177032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:02.071104 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.070981    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:04.070963627 +0000 UTC m=+6.158212928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:02.399021 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.398887    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:01 +0000 UTC" deadline="2027-12-10 06:03:30.345868686 +0000 UTC"
Jul 09 18:03:02.399021 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.398932    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12444h0m27.946941813s"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.477416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.477442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.477562    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:02.477667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.477925    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:02.479202 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:02.478004    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:03.985601 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:03.985453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:03.986109 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:03.985671    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:03.986109 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:03.985707    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:03.986109 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:03.985722    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:03.986109 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:03.985795    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:07.985775972 +0000 UTC m=+10.073025259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:04.086507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:04.086609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.086763    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.086772    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.086832    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:08.086813323 +0000 UTC m=+10.174062608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:04.086878 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.086851    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:08.086841702 +0000 UTC m=+10.174090986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:04.477405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.477541    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:04.478061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.478168    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:04.478222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:04.478418 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:04.478303    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:06.477100 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:06.476977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:06.477100 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:06.477017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:06.477710 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:06.477121    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:06.477710 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:06.476977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:06.477710 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:06.477225    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:06.477710 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:06.477337    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:08.019148 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.019100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:08.019686 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.019308    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:08.019686 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.019331    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:08.019686 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.019342    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:08.019686 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.019404    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:16.019383773 +0000 UTC m=+18.106633079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:08.120446 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.120409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:08.120640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.120468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:08.120640 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.120580    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:08.120640 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.120602    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:08.120790 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.120651    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:16.120636704 +0000 UTC m=+18.207886003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:08.120790 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.120669    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:16.120661588 +0000 UTC m=+18.207910871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:08.480334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.480242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:08.480509 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.480416    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:08.480811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.480667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:08.480952 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.480819    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:08.480952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:08.480672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:08.480952 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:08.480924    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:10.476725 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:10.476649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:10.476725 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:10.476670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:10.477270 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:10.476771    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:10.477270 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:10.476870    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:10.477270 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:10.476935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:10.477270 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:10.477008    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:12.479358 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.479328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:12.479846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.479445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:12.479846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:12.479459    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:12.479846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:12.479577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:12.479846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.479620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:12.479846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:12.479693    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:12.543675 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.543638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjvln" event={"ID":"0fdc1f91-8a06-4af0-95df-82e847e1e57d","Type":"ContainerStarted","Data":"08166eb6f1c0998094c9c9a7be6621ff2340eb3a8309ca71a6a99b8a056d8894"}
Jul 09 18:03:12.545596 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.545557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" event={"ID":"52024e858ee987f467ceba75222ec174","Type":"ContainerStarted","Data":"b38b700736338ba0d1edae081dc8571f923f509273ed108a5c7ce9d4bd8fdb61"}
Jul 09 18:03:12.563452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.563384    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjvln" podStartSLOduration=2.532688907 podStartE2EDuration="13.56336377s" totalImagesPullingTime="11.030674863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:12.562927767 +0000 UTC m=+14.650177071" watchObservedRunningTime="2026-07-09 18:03:12.56336377 +0000 UTC m=+14.650613075"
Jul 09 18:03:12.583367 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:12.583297    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" podStartSLOduration=12.583275202 podStartE2EDuration="12.583275202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:12.582152331 +0000 UTC m=+14.669401636" watchObservedRunningTime="2026-07-09 18:03:12.583275202 +0000 UTC m=+14.670524509"
Jul 09 18:03:13.549205 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.549161    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b88f81944f9da8e5948e6d8adf8450a" containerID="27509bc082a1ba8b20154c69207bc385c9434afd432cf7c10c80b1011d8a05eb" exitCode=0
Jul 09 18:03:13.549673 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.549267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"3b88f81944f9da8e5948e6d8adf8450a","Type":"ContainerDied","Data":"27509bc082a1ba8b20154c69207bc385c9434afd432cf7c10c80b1011d8a05eb"}
Jul 09 18:03:13.551073 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.551048    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="f1efeac351217e85a81ecf67da88b296ac311c1005dec8949f2344fe3f3e1711" exitCode=0
Jul 09 18:03:13.551168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.551078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"f1efeac351217e85a81ecf67da88b296ac311c1005dec8949f2344fe3f3e1711"}
Jul 09 18:03:13.552908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.552880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87jjw" event={"ID":"3022f01d-3e8d-4f25-964b-f0cfbdd07da6","Type":"ContainerStarted","Data":"f10555e60e4b41bea39296b9ea41b60335e16b99b8d2d990b6a4bc3b99ff4c57"}
Jul 09 18:03:13.554327 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.554307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82jsv" event={"ID":"6430784f-f90b-4d10-9461-d80c0d848b40","Type":"ContainerStarted","Data":"8c94b71567bd4eb666f848338003fdadb8dfdc88b29b28587e8c8a8c7886cb3e"}
Jul 09 18:03:13.555650 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.555629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdz76" event={"ID":"44981ba1-e2ac-4bca-83c9-6988a5237880","Type":"ContainerStarted","Data":"be0fd73975fdec904f00865b9dadf15d58193ab20c86c9274e2c75eb9cea5250"}
Jul 09 18:03:13.557002 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.556977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7vxc" event={"ID":"65f8c8c3-8000-4495-bebc-921e1f1a5a9b","Type":"ContainerStarted","Data":"ebf62276ffcceb935d973e9ca6baaca1bc2ae1ee6180fc9c5732d7a0d8cc9458"}
Jul 09 18:03:13.558296 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.558274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l" event={"ID":"dd0ef01d-cc84-452f-8e57-1d72aff552fc","Type":"ContainerStarted","Data":"c86afb24d09905911456c2c9fd69fce9743e7e440b3ca1d7e80830775563217c"}
Jul 09 18:03:13.580402 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.580352    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-87jjw" podStartSLOduration=3.546242674 podStartE2EDuration="14.580335965s" totalImagesPullingTime="11.034093291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:13.58010625 +0000 UTC m=+15.667355555" watchObservedRunningTime="2026-07-09 18:03:13.580335965 +0000 UTC m=+15.667585270"
Jul 09 18:03:13.594514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.594467    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h7vxc" podStartSLOduration=3.567146637 podStartE2EDuration="14.594447209s" totalImagesPullingTime="11.027300572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:13.59416349 +0000 UTC m=+15.681412794" watchObservedRunningTime="2026-07-09 18:03:13.594447209 +0000 UTC m=+15.681696514"
Jul 09 18:03:13.631279 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.630961    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jdz76" podStartSLOduration=3.568743875 podStartE2EDuration="14.630938775s" totalImagesPullingTime="11.0621949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:13.630418238 +0000 UTC m=+15.717667542" watchObservedRunningTime="2026-07-09 18:03:13.630938775 +0000 UTC m=+15.718188080"
Jul 09 18:03:13.648071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:13.648021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-82jsv" podStartSLOduration=3.6578917300000002 podStartE2EDuration="14.647999351s" totalImagesPullingTime="10.990107621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:13.647751207 +0000 UTC m=+15.735000513" watchObservedRunningTime="2026-07-09 18:03:13.647999351 +0000 UTC m=+15.735248656"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.476773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.476773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.476916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:14.476911    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:14.477040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:14.477295 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:14.477130    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:14.562898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.562848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"3b88f81944f9da8e5948e6d8adf8450a","Type":"ContainerStarted","Data":"359b5582e60ecaa0c4d907d10b42cbe5e42a797016a39baeecfc86f62804e906"}
Jul 09 18:03:14.580900 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.580835    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" podStartSLOduration=14.580815551 podStartE2EDuration="14.580815551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:14.580401849 +0000 UTC m=+16.667651159" watchObservedRunningTime="2026-07-09 18:03:14.580815551 +0000 UTC m=+16.668064836"
Jul 09 18:03:14.827861 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:14.827777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:15.253266 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:15.253210    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:03:15.411469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:15.411296    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:03:15.253243646Z","UUID":"d261aa93-4023-497b-a87e-269e8a937723","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:03:15.413446 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:15.413419    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:03:15.413678 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:15.413468    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:03:15.567881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:15.567842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l" event={"ID":"dd0ef01d-cc84-452f-8e57-1d72aff552fc","Type":"ContainerStarted","Data":"088dba53cb4826b568d5c9fda5aeeffb789b7de66c6a035eb82edd0cc821d458"}
Jul 09 18:03:16.085303 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.085261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:16.085499 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.085481    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:16.085581 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.085502    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:16.085581 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.085516    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:16.085680 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.085579    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:32.085556945 +0000 UTC m=+34.172806230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:16.186674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.186629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:16.186853 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.186702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:16.186853 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.186829    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:16.186974 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.186920    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:32.186904509 +0000 UTC m=+34.274153814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:16.186974 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.186840    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:16.187089 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.186989    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:32.186976141 +0000 UTC m=+34.274225441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:16.477028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.476935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:16.477028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.476980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:16.477028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:16.476945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:16.477316 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.477083    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:16.477316 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.477200    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:16.477316 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:16.477303    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:18.477074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.477025    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:18.477740 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:18.477132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:18.477740 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.477140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:18.477740 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:18.477250    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:18.477740 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.477279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:18.477740 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:18.477351    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:18.552489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.552441    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:18.553496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.553473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:18.574954 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:18.574904    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-87jjw"
Jul 09 18:03:20.476842 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:20.476582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:20.477272 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:20.476582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:20.477272 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:20.476957    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:20.477272 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:20.476590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:20.477272 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:20.477051    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:20.477272 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:20.477109    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:22.479627 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:22.479596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:22.480323 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:22.479733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:22.480323 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:22.479744    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:22.480323 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:22.479848    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:22.480323 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:22.479868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:22.480323 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:22.479969    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:24.476133 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:24.476100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:24.476605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:24.476100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:24.476605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:24.476104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:24.476605 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:24.476321    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:24.476605 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:24.476380    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:24.476605 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:24.476274    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:26.476356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.476056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:26.476356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.476058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:26.476356 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:26.476154    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:26.476356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.476060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:26.476356 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:26.476293    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:26.476757 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:26.476367    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:26.591428 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.591397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l" event={"ID":"dd0ef01d-cc84-452f-8e57-1d72aff552fc","Type":"ContainerStarted","Data":"8f6f4b2732a55d697441766da627ee40144c62217444d14e002f5e7ffc384644"}
Jul 09 18:03:26.593389 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:03:26.593680 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593662    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd9e7bc5-49c5-4535-b31c-b9f2b594e74f" containerID="86d61c24af709940ef895346338dfdfa292dfb0e5f1e2b1788fbe33850dc423e" exitCode=1
Jul 09 18:03:26.593799 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"6cc4462526e3c3a574453df7c61708a8aef0794e84727d404f7678b9571dd19d"}
Jul 09 18:03:26.593799 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"b454f599d04a70ec8684cef1a41b65fed002cb98a66275d6f2904e1ef850591a"}
Jul 09 18:03:26.593886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerDied","Data":"86d61c24af709940ef895346338dfdfa292dfb0e5f1e2b1788fbe33850dc423e"}
Jul 09 18:03:26.593886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.593860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"b0fd2a4c3671347be2ae399bac4cb67f18e8e29dec00f31ad014b04ffa2355f0"}
Jul 09 18:03:26.595483 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.595462    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="772e4a1448d016bcdbddd28926a171212f36a01548b5ebb0f5ae90778f3e48cb" exitCode=0
Jul 09 18:03:26.595569 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.595536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"772e4a1448d016bcdbddd28926a171212f36a01548b5ebb0f5ae90778f3e48cb"}
Jul 09 18:03:26.598755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.598559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwftt" event={"ID":"6b0a56a1-54a9-4456-837a-a7ae772a9716","Type":"ContainerStarted","Data":"737bdd9c4d426cadaeadef004e37fa2d73cfaf1121e736809bdfa8838c2ba5a2"}
Jul 09 18:03:26.610254 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.610199    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4n2l" podStartSLOduration=3.35509226 podStartE2EDuration="27.610186057s" totalImagesPullingTime="24.255093797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:26.608934577 +0000 UTC m=+28.696183881" watchObservedRunningTime="2026-07-09 18:03:26.610186057 +0000 UTC m=+28.697435380"
Jul 09 18:03:26.626960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:26.626920    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zwftt" podStartSLOduration=2.212480221 podStartE2EDuration="27.626895737s" totalImagesPullingTime="25.414415516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:26.626761247 +0000 UTC m=+28.714010562" watchObservedRunningTime="2026-07-09 18:03:26.626895737 +0000 UTC m=+28.714145040"
Jul 09 18:03:27.604182 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:27.604114    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:03:27.604750 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:27.604488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"c7b46fef0a5d795d8eff831517a49fac7c09a2fb7a11c87231e1b066e6d29cf8"}
Jul 09 18:03:27.604750 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:27.604604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"7b80773494faf9c4ba08db7515d25dc08280cc1fe576a898ceabe747f3c66e49"}
Jul 09 18:03:28.477272 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:28.477223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:28.477445 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:28.477330    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:28.477445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:28.477344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:28.477445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:28.477376    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:28.480325 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:28.477822    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:28.480325 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:28.477965    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:28.608482 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:28.608452    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="ea20a34ca5ae3063c97fb3a942a8dc141f7f18f1131d4474133341b82cb4c9b0" exitCode=0
Jul 09 18:03:28.608482 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:28.608492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"ea20a34ca5ae3063c97fb3a942a8dc141f7f18f1131d4474133341b82cb4c9b0"}
Jul 09 18:03:29.614276 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:29.614248    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:03:29.614721 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:29.614640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"b9f96516cd327b0e3abaf0e08bcf17daff68715f51eeef8f428c497cd33058b9"}
Jul 09 18:03:30.476830 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:30.476759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:30.476986 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:30.476757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:30.476986 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:30.476867    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:30.476986 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:30.476757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:30.476986 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:30.476972    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:30.477182 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:30.477011    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:30.619156 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:30.619125    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="ebf6f41a535ffaa1fbf9f20022012a96f9c3ab91d5df6ca1e571bc00b6126e79" exitCode=0
Jul 09 18:03:30.619543 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:30.619162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"ebf6f41a535ffaa1fbf9f20022012a96f9c3ab91d5df6ca1e571bc00b6126e79"}
Jul 09 18:03:31.625557 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.625369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:03:31.626139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.625993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"699a7aae5b7f9fa3dc4c3e5344888d625b051fc33923b64f18077b095ec5659a"}
Jul 09 18:03:31.626418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.626259    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:31.626418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.626322    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:31.626418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.626364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:31.626775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.626757    2565 scope.go:117] "RemoveContainer" containerID="86d61c24af709940ef895346338dfdfa292dfb0e5f1e2b1788fbe33850dc423e"
Jul 09 18:03:31.647078 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.647052    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:31.647217 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:31.647199    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:03:32.104854 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.104766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:32.104997 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.104915    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:32.104997 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.104938    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:32.104997 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.104956    2565 projected.go:196] Error preparing data for projected volume kube-api-access-g6ql4 for pod openshift-network-diagnostics/network-check-target-njgpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:32.105102 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.105019    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4 podName:35f8a758-2b3d-4c30-9ba0-f946a933e4f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:04.105003189 +0000 UTC m=+66.192252471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6ql4" (UniqueName: "kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4") pod "network-check-target-njgpn" (UID: "35f8a758-2b3d-4c30-9ba0-f946a933e4f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:32.205537 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.205501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:32.205707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.205557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:32.205707 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.205662    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:32.205805 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.205718    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:04.205703673 +0000 UTC m=+66.292952954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:32.205805 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.205661    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:32.205903 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.205835    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret podName:24b38b34-371d-43da-9ed4-6da5eb6ee973 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:04.205811932 +0000 UTC m=+66.293061220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret") pod "global-pull-secret-syncer-kmbmg" (UID: "24b38b34-371d-43da-9ed4-6da5eb6ee973") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:32.476438 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.476334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:32.476588 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.476342    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:32.476588 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.476455    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:32.476588 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.476492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:32.476758 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.476593    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:32.476758 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:32.476681    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:32.633114 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.633090    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:03:32.633632 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.633514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" event={"ID":"dd9e7bc5-49c5-4535-b31c-b9f2b594e74f","Type":"ContainerStarted","Data":"162dc211a15a5de56808a2d0131ad17d3891faca8cd81e2f49e377357ad04fcc"}
Jul 09 18:03:32.662396 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:32.662348    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn" podStartSLOduration=8.219489232 podStartE2EDuration="33.662331861s" totalImagesPullingTime="25.442842629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:32.662012575 +0000 UTC m=+34.749261892" watchObservedRunningTime="2026-07-09 18:03:32.662331861 +0000 UTC m=+34.749581164"
Jul 09 18:03:33.289367 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.289326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmbmg"]
Jul 09 18:03:33.289537 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.289502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:33.289645 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:33.289617    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:33.292689 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.292662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjjkk"]
Jul 09 18:03:33.292839 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.292782    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:33.292920 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:33.292892    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:33.293328 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.293304    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njgpn"]
Jul 09 18:03:33.293408 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:33.293399    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:33.293529 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:33.293488    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:34.476490 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:34.476454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:34.476911 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:34.476588    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:35.476905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:35.476873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:35.477211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:35.476906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:35.477211 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:35.477008    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:35.477211 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:35.477116    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:35.644267 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:35.644174    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="aade730b468e5765e9ca27848986b66de1c4c73d4be9a5d6db035ade0c868d71" exitCode=0
Jul 09 18:03:35.644267 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:35.644227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"aade730b468e5765e9ca27848986b66de1c4c73d4be9a5d6db035ade0c868d71"}
Jul 09 18:03:36.476470 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:36.476294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:36.476620 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:36.476556    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:36.649858 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:36.649824    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df6f073-d9d4-4339-8fa3-8605f157a4e2" containerID="62d95fc0ba0c012acc58ce18aba7d27a6e1bde6dde667c558a696ca4bc3e51f7" exitCode=0
Jul 09 18:03:36.650459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:36.649888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerDied","Data":"62d95fc0ba0c012acc58ce18aba7d27a6e1bde6dde667c558a696ca4bc3e51f7"}
Jul 09 18:03:37.476459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:37.476427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:37.476459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:37.476442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:37.476659 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:37.476531    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmbmg" podUID="24b38b34-371d-43da-9ed4-6da5eb6ee973"
Jul 09 18:03:37.476659 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:37.476596    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjjkk" podUID="8d46ab2f-e328-4fc2-9c11-9ef297a01f63"
Jul 09 18:03:37.655569 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:37.655530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbckn" event={"ID":"4df6f073-d9d4-4339-8fa3-8605f157a4e2","Type":"ContainerStarted","Data":"c940c9f9a1829bfa8aa7275c24c727276713057c0d22b404944acce0182cb6f7"}
Jul 09 18:03:37.679866 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:37.679806    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cbckn" podStartSLOduration=7.695150515 podStartE2EDuration="38.679787394s" totalImagesPullingTime="30.984636879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:37.679186472 +0000 UTC m=+39.766435776" watchObservedRunningTime="2026-07-09 18:03:37.679787394 +0000 UTC m=+39.767036698"
Jul 09 18:03:38.478452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.478404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:38.478733 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.478594    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njgpn" podUID="35f8a758-2b3d-4c30-9ba0-f946a933e4f1"
Jul 09 18:03:38.633971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.633949    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeReady"
Jul 09 18:03:38.634099 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.634076    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:03:38.668776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.668744    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:03:38.680885 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.680861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.682641 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.682613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:03:38.683510 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.683487    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:03:38.683510 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.683491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:03:38.683719 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.683498    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:03:38.683836 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.683822    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pxfrt\""
Jul 09 18:03:38.685462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.685443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qgpt5"]
Jul 09 18:03:38.690407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.690390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:03:38.690914 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.690899    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zt2s9"]
Jul 09 18:03:38.691026 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.691011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.693831 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.693808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:03:38.693926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.693853    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qqkr6\""
Jul 09 18:03:38.693926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.693912    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:03:38.694015 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.694006    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:03:38.697056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.697041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.698868 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.698825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgpt5"]
Jul 09 18:03:38.699455 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.699435    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ln8j9\""
Jul 09 18:03:38.699546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.699500    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt2s9"]
Jul 09 18:03:38.699661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.699644    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:03:38.699839 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.699822    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:03:38.859876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.859876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.859876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpjw\" (UniqueName: \"kubernetes.io/projected/1d432f1b-b963-4d1a-bec9-052434b9454d-kube-api-access-bbpjw\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0af21ee9-3d83-4aa9-a541-ddb07176e777-cert\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d432f1b-b963-4d1a-bec9-052434b9454d-config-volume\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.859993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d432f1b-b963-4d1a-bec9-052434b9454d-tmp-dir\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.860087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8txt\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.860329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.860159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szhh\" (UniqueName: \"kubernetes.io/projected/0af21ee9-3d83-4aa9-a541-ddb07176e777-kube-api-access-4szhh\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.961116 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961116 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961323 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961323 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8txt\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szhh\" (UniqueName: \"kubernetes.io/projected/0af21ee9-3d83-4aa9-a541-ddb07176e777-kube-api-access-4szhh\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.961395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpjw\" (UniqueName: \"kubernetes.io/projected/1d432f1b-b963-4d1a-bec9-052434b9454d-kube-api-access-bbpjw\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.961647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.961647 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.961526    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:38.961647 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.961544    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:38.961647 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.961624    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:39.461596381 +0000 UTC m=+41.548845683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:38.961887 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0af21ee9-3d83-4aa9-a541-ddb07176e777-cert\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.961942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d432f1b-b963-4d1a-bec9-052434b9454d-config-volume\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.961995 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.961995 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.961985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d432f1b-b963-4d1a-bec9-052434b9454d-tmp-dir\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.962270 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.962218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.962361 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.962294    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:38.962361 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:38.962350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:39.462334511 +0000 UTC m=+41.549583844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:38.962464 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.962386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d432f1b-b963-4d1a-bec9-052434b9454d-tmp-dir\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.962653 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.962631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d432f1b-b963-4d1a-bec9-052434b9454d-config-volume\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.963006 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.962984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.965642 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.965618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0af21ee9-3d83-4aa9-a541-ddb07176e777-cert\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.965723 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.965641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.965723 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.965656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.970342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.970322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.970693 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.970664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpjw\" (UniqueName: \"kubernetes.io/projected/1d432f1b-b963-4d1a-bec9-052434b9454d-kube-api-access-bbpjw\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:38.971186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.971168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8txt\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:38.971671 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.971652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szhh\" (UniqueName: \"kubernetes.io/projected/0af21ee9-3d83-4aa9-a541-ddb07176e777-kube-api-access-4szhh\") pod \"ingress-canary-qgpt5\" (UID: \"0af21ee9-3d83-4aa9-a541-ddb07176e777\") " pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:38.999114 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:38.999088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgpt5"
Jul 09 18:03:39.202480 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.202252    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgpt5"]
Jul 09 18:03:39.207269 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:39.207226    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af21ee9_3d83_4aa9_a541_ddb07176e777.slice/crio-d5ccecfafe236ae63595606ff2d7ffe692d521391c8541bd5a44c580d45d5c53 WatchSource:0}: Error finding container d5ccecfafe236ae63595606ff2d7ffe692d521391c8541bd5a44c580d45d5c53: Status 404 returned error can't find the container with id d5ccecfafe236ae63595606ff2d7ffe692d521391c8541bd5a44c580d45d5c53
Jul 09 18:03:39.466067 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.466031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:39.466281 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.466096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:39.466281 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:39.466177    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:39.466281 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:39.466266    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:40.466226362 +0000 UTC m=+42.553475662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:39.466281 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:39.466179    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:39.466281 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:39.466283    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:39.466448 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:39.466307    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:40.466301152 +0000 UTC m=+42.553550469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:39.476807 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.476787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:03:39.476895 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.476818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:03:39.478996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.478963    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:03:39.478996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.478986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:03:39.479156 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.479029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmkm4\""
Jul 09 18:03:39.661181 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:39.661148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgpt5" event={"ID":"0af21ee9-3d83-4aa9-a541-ddb07176e777","Type":"ContainerStarted","Data":"d5ccecfafe236ae63595606ff2d7ffe692d521391c8541bd5a44c580d45d5c53"}
Jul 09 18:03:40.474531 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.474497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.474561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:40.474696    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:40.474726    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:40.474747    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:40.474796    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:42.474774544 +0000 UTC m=+44.562023829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:40.475093 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:40.474817    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:42.474807264 +0000 UTC m=+44.562056548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:40.476625 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.476603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:03:40.479970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.479944    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hzqhq\""
Jul 09 18:03:40.479970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.479944    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:03:40.480164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:40.479945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:03:41.478774 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:41.478724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:41.583272 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:41.583223    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:41.681734 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:41.681687    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:42.491494 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:42.491468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:42.491519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:42.491603    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:42.491619    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:42.491620    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:42.491671    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:46.491653933 +0000 UTC m=+48.578903218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:42.491862 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:42.491688    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:46.491682232 +0000 UTC m=+48.578931566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:42.671324 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:42.671289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgpt5" event={"ID":"0af21ee9-3d83-4aa9-a541-ddb07176e777","Type":"ContainerStarted","Data":"452f8ebc0f5f661902592b9a6e3c231ea645503d7736722468287bbd63ab08bc"}
Jul 09 18:03:42.690141 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:42.690092    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qgpt5" podStartSLOduration=1.491186115 podStartE2EDuration="4.690077496s" totalImagesPullingTime="3.198891381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:42.689986386 +0000 UTC m=+44.777235690" watchObservedRunningTime="2026-07-09 18:03:42.690077496 +0000 UTC m=+44.777326806"
Jul 09 18:03:46.056107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:46.056074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:46.117732 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:46.117695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:46.521894 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:46.521853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:46.522073 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:46.521947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:46.522073 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:46.522008    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:46.522073 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:46.522061    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:46.522073 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:46.522072    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:54.522056101 +0000 UTC m=+56.609305383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:46.522073 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:46.522075    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:46.522288 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:46.522124    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:54.522108299 +0000 UTC m=+56.609357587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:48.270360 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:48.270325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:48.347292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:48.347256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:48.422322 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:48.422288    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:49.215246 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.215199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:49.293857 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.293817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:49.531561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.531479    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"]
Jul 09 18:03:49.543322 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.543294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"
Jul 09 18:03:49.545369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.545345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"]
Jul 09 18:03:49.546448 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.546424    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.546573 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.546497    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.547393 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.547364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wk95d\""
Jul 09 18:03:49.578644 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.578621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-89h9v"]
Jul 09 18:03:49.594852 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.594832    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.597625 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.597603    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 18:03:49.597751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.597655    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.597751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.597715    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.597940 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.597924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 18:03:49.598302 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.598283    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l49db\""
Jul 09 18:03:49.599669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.599653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-89h9v"]
Jul 09 18:03:49.605587 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.605569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 18:03:49.644847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.644816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hghw\" (UniqueName: \"kubernetes.io/projected/ab136634-580b-4ce5-80fa-6c52e0e7e747-kube-api-access-6hghw\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.644971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.644877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-trusted-ca\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.644971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.644921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab136634-580b-4ce5-80fa-6c52e0e7e747-serving-cert\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.644971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.644943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zb8\" (UniqueName: \"kubernetes.io/projected/d0f5cfa2-5396-418f-b708-79e38e746ac4-kube-api-access-48zb8\") pod \"volume-data-source-validator-78495597f6-pqf7p\" (UID: \"d0f5cfa2-5396-418f-b708-79e38e746ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"
Jul 09 18:03:49.644971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.644961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-config\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.682669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.682633    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"]
Jul 09 18:03:49.706852 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.706823    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"]
Jul 09 18:03:49.706978 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.706940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.709837 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.709789    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 09 18:03:49.709996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.709873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-24rmh\""
Jul 09 18:03:49.710065 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.709888    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.710065 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.710014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.722886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.722813    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"]
Jul 09 18:03:49.723027 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.722937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.726142 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.726106    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 09 18:03:49.726308 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.726154    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 09 18:03:49.726308 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.726165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.726308 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.726299    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.726752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.726735    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qr8j7\""
Jul 09 18:03:49.745137 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745113    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x6k6c"]
Jul 09 18:03:49.745262 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"
Jul 09 18:03:49.745360 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-trusted-ca\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.745416 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.745474 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab136634-580b-4ce5-80fa-6c52e0e7e747-serving-cert\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.745530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zb8\" (UniqueName: \"kubernetes.io/projected/d0f5cfa2-5396-418f-b708-79e38e746ac4-kube-api-access-48zb8\") pod \"volume-data-source-validator-78495597f6-pqf7p\" (UID: \"d0f5cfa2-5396-418f-b708-79e38e746ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"
Jul 09 18:03:49.745530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-config\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.745627 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pz46\" (UniqueName: \"kubernetes.io/projected/bc24e4ed-e41e-4cd5-b979-1760c229d004-kube-api-access-4pz46\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.745627 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.745627 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtcm\" (UniqueName: \"kubernetes.io/projected/5073d86a-582f-4f04-ad5a-130ea337c439-kube-api-access-9dtcm\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.745627 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hghw\" (UniqueName: \"kubernetes.io/projected/ab136634-580b-4ce5-80fa-6c52e0e7e747-kube-api-access-6hghw\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.745810 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.745640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5073d86a-582f-4f04-ad5a-130ea337c439-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.746263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.746225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-config\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.747772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.747748    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5ppnb\""
Jul 09 18:03:49.747876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.747857    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab136634-580b-4ce5-80fa-6c52e0e7e747-trusted-ca\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.753117 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.753035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab136634-580b-4ce5-80fa-6c52e0e7e747-serving-cert\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.760469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.760447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zb8\" (UniqueName: \"kubernetes.io/projected/d0f5cfa2-5396-418f-b708-79e38e746ac4-kube-api-access-48zb8\") pod \"volume-data-source-validator-78495597f6-pqf7p\" (UID: \"d0f5cfa2-5396-418f-b708-79e38e746ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"
Jul 09 18:03:49.760559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.760505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hghw\" (UniqueName: \"kubernetes.io/projected/ab136634-580b-4ce5-80fa-6c52e0e7e747-kube-api-access-6hghw\") pod \"console-operator-7866497494-89h9v\" (UID: \"ab136634-580b-4ce5-80fa-6c52e0e7e747\") " pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.765437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.765165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"]
Jul 09 18:03:49.765437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.765211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"]
Jul 09 18:03:49.765437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.765249    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"]
Jul 09 18:03:49.765437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.765272    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x6k6c"]
Jul 09 18:03:49.765437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.765367    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.783465 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.782822    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.783465 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.783140    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.784083 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.784060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 09 18:03:49.793534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.788597    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-zml89"]
Jul 09 18:03:49.793534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.790368    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-swpzj\""
Jul 09 18:03:49.793534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.790595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 09 18:03:49.795335 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.795310    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 09 18:03:49.814707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.814681    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"]
Jul 09 18:03:49.814812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.814752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.819704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.819682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 18:03:49.819839 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.819725    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f92d9\""
Jul 09 18:03:49.819913 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.819868    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 18:03:49.828828 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.828801    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6fd9c96588-sbl55"]
Jul 09 18:03:49.828941 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.828905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.831785 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.831761    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.831906 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.831849    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 18:03:49.831906 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.831867    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.832060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.832044    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sr9dv\""
Jul 09 18:03:49.833511 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.833483    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 18:03:49.846450 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdz7q\" (UniqueName: \"kubernetes.io/projected/2dc89993-a760-495b-9819-6307bcf51561-kube-api-access-wdz7q\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.846559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2dc89993-a760-495b-9819-6307bcf51561-config\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.846559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89993-a760-495b-9819-6307bcf51561-serving-cert\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.846559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtcm\" (UniqueName: \"kubernetes.io/projected/5073d86a-582f-4f04-ad5a-130ea337c439-kube-api-access-9dtcm\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.846717 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-serving-cert\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.846717 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rt7t\" (UniqueName: \"kubernetes.io/projected/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-kube-api-access-4rt7t\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.846717 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pz46\" (UniqueName: \"kubernetes.io/projected/bc24e4ed-e41e-4cd5-b979-1760c229d004-kube-api-access-4pz46\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.846859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.846859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-snapshots\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.846859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-tmp\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.846859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.846859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e9a8c762-322e-4594-b759-311e8ed6b8ee-nginx-conf\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846889    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"]
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.846923    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.846951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6n6\" (UniqueName: \"kubernetes.io/projected/f5618d41-a7bd-4c70-9131-b48ed11251ab-kube-api-access-sz6n6\") pod \"network-check-source-869dcfcdcd-7dm27\" (UID: \"f5618d41-a7bd-4c70-9131-b48ed11251ab\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.846984    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.346968124 +0000 UTC m=+52.434217409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847020    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.847061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.847356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.847356 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.847114    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:49.847356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5073d86a-582f-4f04-ad5a-130ea337c439-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.847356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.847356 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.847165    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.347148976 +0000 UTC m=+52.434398270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:49.847878 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.847861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5073d86a-582f-4f04-ad5a-130ea337c439-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.852575 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.852551    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 18:03:49.852676 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.852641    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.852729 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.852697    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.852789 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.852773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 18:03:49.853776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.853761    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"
Jul 09 18:03:49.853865 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.853851    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 18:03:49.853986 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.853961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 18:03:49.854060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.854045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4whfj\""
Jul 09 18:03:49.859911 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.859885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pz46\" (UniqueName: \"kubernetes.io/projected/bc24e4ed-e41e-4cd5-b979-1760c229d004-kube-api-access-4pz46\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:49.860053 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.860035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtcm\" (UniqueName: \"kubernetes.io/projected/5073d86a-582f-4f04-ad5a-130ea337c439-kube-api-access-9dtcm\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:49.871184 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.871165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"]
Jul 09 18:03:49.871350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.871194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-zml89"]
Jul 09 18:03:49.871350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.871208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fd9c96588-sbl55"]
Jul 09 18:03:49.871350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.871220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"]
Jul 09 18:03:49.871350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.871304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:49.873836 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.873818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 18:03:49.873930 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.873883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 18:03:49.874035 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.874020    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.874035 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.874029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.874288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.874274    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9p5xd\""
Jul 09 18:03:49.904989 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.904967    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:49.947559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.947559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-snapshots\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5b51860-1dbb-4ff1-a7cb-0273f891f537-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-tmp\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e9a8c762-322e-4594-b759-311e8ed6b8ee-nginx-conf\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6n6\" (UniqueName: \"kubernetes.io/projected/f5618d41-a7bd-4c70-9131-b48ed11251ab-kube-api-access-sz6n6\") pod \"network-check-source-869dcfcdcd-7dm27\" (UID: \"f5618d41-a7bd-4c70-9131-b48ed11251ab\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.947975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdz7q\" (UniqueName: \"kubernetes.io/projected/2dc89993-a760-495b-9819-6307bcf51561-kube-api-access-wdz7q\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2dc89993-a760-495b-9819-6307bcf51561-config\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89993-a760-495b-9819-6307bcf51561-serving-cert\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvcgc\" (UniqueName: \"kubernetes.io/projected/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-kube-api-access-pvcgc\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b51860-1dbb-4ff1-a7cb-0273f891f537-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.948188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-serving-cert\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-snapshots\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rt7t\" (UniqueName: \"kubernetes.io/projected/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-kube-api-access-4rt7t\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log" limit=1024
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-default-certificate\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-stats-auth\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwt7\" (UniqueName: \"kubernetes.io/projected/b5b51860-1dbb-4ff1-a7cb-0273f891f537-kube-api-access-hqwt7\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.948501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-tmp\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.948585    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:03:49.949151 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:49.948655    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.448615753 +0000 UTC m=+52.535865036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:03:49.951263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.950374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.951263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.950641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2dc89993-a760-495b-9819-6307bcf51561-config\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.951263 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.950781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e9a8c762-322e-4594-b759-311e8ed6b8ee-nginx-conf\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:49.951743 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.951682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2dc89993-a760-495b-9819-6307bcf51561-serving-cert\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.954757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.951986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-serving-cert\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:49.966558 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.966530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6n6\" (UniqueName: \"kubernetes.io/projected/f5618d41-a7bd-4c70-9131-b48ed11251ab-kube-api-access-sz6n6\") pod \"network-check-source-869dcfcdcd-7dm27\" (UID: \"f5618d41-a7bd-4c70-9131-b48ed11251ab\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"
Jul 09 18:03:49.966774 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.966750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdz7q\" (UniqueName: \"kubernetes.io/projected/2dc89993-a760-495b-9819-6307bcf51561-kube-api-access-wdz7q\") pod \"service-ca-operator-7695fc6c8c-jt9jc\" (UID: \"2dc89993-a760-495b-9819-6307bcf51561\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:49.966839 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:49.966751    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rt7t\" (UniqueName: \"kubernetes.io/projected/44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3-kube-api-access-4rt7t\") pod \"insights-operator-68d9dd94cf-x6k6c\" (UID: \"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3\") " pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:50.019848 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.019819    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p"]
Jul 09 18:03:50.025563 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.025532    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f5cfa2_5396_418f_b708_79e38e746ac4.slice/crio-d4671d69ea4ce77e8ca0048883ea8d42790193c65bed1224142768c382c48cf8 WatchSource:0}: Error finding container d4671d69ea4ce77e8ca0048883ea8d42790193c65bed1224142768c382c48cf8: Status 404 returned error can't find the container with id d4671d69ea4ce77e8ca0048883ea8d42790193c65bed1224142768c382c48cf8
Jul 09 18:03:50.049605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.049605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.049605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvcgc\" (UniqueName: \"kubernetes.io/projected/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-kube-api-access-pvcgc\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.049846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.049686    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:03:50.049846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.049731    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.549711521 +0000 UTC m=+52.636960804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:03:50.049846 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.049756    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.549748756 +0000 UTC m=+52.636998039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:03:50.049846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b51860-1dbb-4ff1-a7cb-0273f891f537-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.049846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-default-certificate\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.050103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-stats-auth\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.050103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwt7\" (UniqueName: \"kubernetes.io/projected/b5b51860-1dbb-4ff1-a7cb-0273f891f537-kube-api-access-hqwt7\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.050103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.049922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5b51860-1dbb-4ff1-a7cb-0273f891f537-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.050438 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.050418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5b51860-1dbb-4ff1-a7cb-0273f891f537-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.052100 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.052071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b51860-1dbb-4ff1-a7cb-0273f891f537-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.052216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.052200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-stats-auth\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.052316 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.052301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-default-certificate\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.058391 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.058374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvcgc\" (UniqueName: \"kubernetes.io/projected/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-kube-api-access-pvcgc\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.059258 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.059226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"
Jul 09 18:03:50.064026 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.064004    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-89h9v"]
Jul 09 18:03:50.065865 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.065845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwt7\" (UniqueName: \"kubernetes.io/projected/b5b51860-1dbb-4ff1-a7cb-0273f891f537-kube-api-access-hqwt7\") pod \"kube-storage-version-migrator-operator-799cb7f958-f5qgx\" (UID: \"b5b51860-1dbb-4ff1-a7cb-0273f891f537\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.066574 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.066555    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab136634_580b_4ce5_80fa_6c52e0e7e747.slice/crio-ba6a2ac5d5d598990593bf51c665a26b5049a3395d6d622d11883862b055cd66 WatchSource:0}: Error finding container ba6a2ac5d5d598990593bf51c665a26b5049a3395d6d622d11883862b055cd66: Status 404 returned error can't find the container with id ba6a2ac5d5d598990593bf51c665a26b5049a3395d6d622d11883862b055cd66
Jul 09 18:03:50.099833 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.099807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c"
Jul 09 18:03:50.138164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.138138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"
Jul 09 18:03:50.179991 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.179754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"
Jul 09 18:03:50.225889 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.225798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27"]
Jul 09 18:03:50.232564 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.232515    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5618d41_a7bd_4c70_9131_b48ed11251ab.slice/crio-aec43af5abf5c6727f4e3ba37f7b81f58a525eaecdb93d9cdf70df29cc3ebcf6 WatchSource:0}: Error finding container aec43af5abf5c6727f4e3ba37f7b81f58a525eaecdb93d9cdf70df29cc3ebcf6: Status 404 returned error can't find the container with id aec43af5abf5c6727f4e3ba37f7b81f58a525eaecdb93d9cdf70df29cc3ebcf6
Jul 09 18:03:50.256161 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.256000    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x6k6c"]
Jul 09 18:03:50.260163 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.260128    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c78c0b_6eaf_4d46_8ae4_3e3544b25fa3.slice/crio-a80821c67843a680879068335b7e9f56bcd92e89a23be8b276237ffafb11b694 WatchSource:0}: Error finding container a80821c67843a680879068335b7e9f56bcd92e89a23be8b276237ffafb11b694: Status 404 returned error can't find the container with id a80821c67843a680879068335b7e9f56bcd92e89a23be8b276237ffafb11b694
Jul 09 18:03:50.307350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.307176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc"]
Jul 09 18:03:50.310024 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.309998    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc89993_a760_495b_9819_6307bcf51561.slice/crio-bde6312720bb3895fadbae503633522548fec7e8ededfddff81c0133bc08cf97 WatchSource:0}: Error finding container bde6312720bb3895fadbae503633522548fec7e8ededfddff81c0133bc08cf97: Status 404 returned error can't find the container with id bde6312720bb3895fadbae503633522548fec7e8ededfddff81c0133bc08cf97
Jul 09 18:03:50.332132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.332093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx"]
Jul 09 18:03:50.335408 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:50.335378    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b51860_1dbb_4ff1_a7cb_0273f891f537.slice/crio-fa42e1b23aa5b36596099cc84916f34d69f7a9b8b9e47ca35c3172b1a95f3e23 WatchSource:0}: Error finding container fa42e1b23aa5b36596099cc84916f34d69f7a9b8b9e47ca35c3172b1a95f3e23: Status 404 returned error can't find the container with id fa42e1b23aa5b36596099cc84916f34d69f7a9b8b9e47ca35c3172b1a95f3e23
Jul 09 18:03:50.352404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.352380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:50.352535 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.352519    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:50.352594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.352551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:50.352594 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.352575    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.352559845 +0000 UTC m=+53.439809128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:50.352705 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.352643    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:03:50.352705 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.352699    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.352681802 +0000 UTC m=+53.439931098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:03:50.453779 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.453744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:50.453914 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.453890    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:03:50.453974 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.453956    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.45393772 +0000 UTC m=+53.541187003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:03:50.555079 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.555045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.555285 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.555088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:50.555285 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.555217    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:03:50.555404 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.555305    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.555288249 +0000 UTC m=+53.642537531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:03:50.555404 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:50.555326    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.555319215 +0000 UTC m=+53.642568496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:03:50.693862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.693828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c" event={"ID":"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3","Type":"ContainerStarted","Data":"a80821c67843a680879068335b7e9f56bcd92e89a23be8b276237ffafb11b694"}
Jul 09 18:03:50.695150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.695120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc" event={"ID":"2dc89993-a760-495b-9819-6307bcf51561","Type":"ContainerStarted","Data":"bde6312720bb3895fadbae503633522548fec7e8ededfddff81c0133bc08cf97"}
Jul 09 18:03:50.696527 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.696495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27" event={"ID":"f5618d41-a7bd-4c70-9131-b48ed11251ab","Type":"ContainerStarted","Data":"aec43af5abf5c6727f4e3ba37f7b81f58a525eaecdb93d9cdf70df29cc3ebcf6"}
Jul 09 18:03:50.697844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.697812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p" event={"ID":"d0f5cfa2-5396-418f-b708-79e38e746ac4","Type":"ContainerStarted","Data":"d4671d69ea4ce77e8ca0048883ea8d42790193c65bed1224142768c382c48cf8"}
Jul 09 18:03:50.699265 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.699224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx" event={"ID":"b5b51860-1dbb-4ff1-a7cb-0273f891f537","Type":"ContainerStarted","Data":"fa42e1b23aa5b36596099cc84916f34d69f7a9b8b9e47ca35c3172b1a95f3e23"}
Jul 09 18:03:50.700468 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:50.700448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-89h9v" event={"ID":"ab136634-580b-4ce5-80fa-6c52e0e7e747","Type":"ContainerStarted","Data":"ba6a2ac5d5d598990593bf51c665a26b5049a3395d6d622d11883862b055cd66"}
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:51.363736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:51.363813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.363993    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.364056    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.364039006 +0000 UTC m=+55.451288294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.364488    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:03:51.364578 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.364540    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.364524576 +0000 UTC m=+55.451773862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:03:51.465709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:51.465092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:51.465709 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.465282    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:03:51.465709 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.465345    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.465326062 +0000 UTC m=+55.552575362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:03:51.566735 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:51.565989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:51.566735 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:51.566058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:51.566735 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.566201    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.566183101 +0000 UTC m=+55.653432384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:03:51.566735 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.566615    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:03:51.566735 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:51.566665    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.566649106 +0000 UTC m=+55.653898392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:03:52.712025 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:52.711883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p" event={"ID":"d0f5cfa2-5396-418f-b708-79e38e746ac4","Type":"ContainerStarted","Data":"65e46e253175da6108137ff436bc739ea787822995d6c7655125c87e92ced4b5"}
Jul 09 18:03:53.386708 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:53.386656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:53.386906 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:53.386746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:53.386906 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.386833    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:03:53.386906 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.386904    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.386887433 +0000 UTC m=+59.474136715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:03:53.387048 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.386898    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:53.387048 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.386946    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.386935763 +0000 UTC m=+59.474185047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:53.487841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:53.487786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:53.488035 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.487979    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:03:53.488113 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.488055    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.488037386 +0000 UTC m=+59.575286670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:03:53.589248 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:53.589198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:53.589433 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:53.589272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:53.589433 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.589354    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:03:53.589433 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.589428    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.589412164 +0000 UTC m=+59.676661446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:03:53.589607 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:53.589443    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.589437192 +0000 UTC m=+59.676686473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:03:54.598293 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:54.598191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:54.598372    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:54.598402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:54.598451    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls podName:1d432f1b-b963-4d1a-bec9-052434b9454d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:10.598433548 +0000 UTC m=+72.685682831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls") pod "dns-default-zt2s9" (UID: "1d432f1b-b963-4d1a-bec9-052434b9454d") : secret "dns-default-metrics-tls" not found
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:54.598532    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:54.598546    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-599847f6ff-wx5zb: secret "image-registry-tls" not found
Jul 09 18:03:54.598763 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:54.598609    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls podName:76f76cc9-2a12-46be-b7e8-fbb3258112a6 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:10.598594127 +0000 UTC m=+72.685843413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls") pod "image-registry-599847f6ff-wx5zb" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6") : secret "image-registry-tls" not found
Jul 09 18:03:55.722965 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.722937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx" event={"ID":"b5b51860-1dbb-4ff1-a7cb-0273f891f537","Type":"ContainerStarted","Data":"64221631b670122787015ad6d057ffa02a49668f54f7be961fd9a884b1badd37"}
Jul 09 18:03:55.725185 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.725164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/0.log"
Jul 09 18:03:55.725309 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.725203    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab136634-580b-4ce5-80fa-6c52e0e7e747" containerID="2b5c4c3bbef0d456c8de7ce922bc149201285232681809c428875280d61d685b" exitCode=255
Jul 09 18:03:55.725356 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.725327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-89h9v" event={"ID":"ab136634-580b-4ce5-80fa-6c52e0e7e747","Type":"ContainerDied","Data":"2b5c4c3bbef0d456c8de7ce922bc149201285232681809c428875280d61d685b"}
Jul 09 18:03:55.725502 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.725485    2565 scope.go:117] "RemoveContainer" containerID="2b5c4c3bbef0d456c8de7ce922bc149201285232681809c428875280d61d685b"
Jul 09 18:03:55.727301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.727225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c" event={"ID":"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3","Type":"ContainerStarted","Data":"b906ed95a1a63d31ae25a3bca1cb889de64489fe74570af45cff874a00eb5e08"}
Jul 09 18:03:55.730066 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.730044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc" event={"ID":"2dc89993-a760-495b-9819-6307bcf51561","Type":"ContainerStarted","Data":"def76b1dd2f1f4fc691db9dc1b476393db66fd1a68dd98eea7dc6ddf6610045a"}
Jul 09 18:03:55.731806 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.731779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27" event={"ID":"f5618d41-a7bd-4c70-9131-b48ed11251ab","Type":"ContainerStarted","Data":"4a31533f6640ea685faba9ff62b9bb1b4230405358bc5b939b888b99312d2ac3"}
Jul 09 18:03:55.741923 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.741628    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pqf7p" podStartSLOduration=4.807780574 podStartE2EDuration="6.741609607s" totalImagesPullingTime="1.933829033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:52.731154717 +0000 UTC m=+54.818404025" watchObservedRunningTime="2026-07-09 18:03:55.741609607 +0000 UTC m=+57.828858913"
Jul 09 18:03:55.757123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.757068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7dm27" podStartSLOduration=1.469153896 podStartE2EDuration="6.757049785s" totalImagesPullingTime="5.287895889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:55.75630633 +0000 UTC m=+57.843555655" watchObservedRunningTime="2026-07-09 18:03:55.757049785 +0000 UTC m=+57.844299089"
Jul 09 18:03:55.757285 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.757227    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx" podStartSLOduration=1.586365572 podStartE2EDuration="6.757217723s" totalImagesPullingTime="5.170852151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:55.741165147 +0000 UTC m=+57.828414452" watchObservedRunningTime="2026-07-09 18:03:55.757217723 +0000 UTC m=+57.844467028"
Jul 09 18:03:55.776247 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.775112    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c" podStartSLOduration=1.530682095 podStartE2EDuration="6.775092127s" totalImagesPullingTime="5.244410032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:55.774209506 +0000 UTC m=+57.861458823" watchObservedRunningTime="2026-07-09 18:03:55.775092127 +0000 UTC m=+57.862341432"
Jul 09 18:03:55.790555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:55.790507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc" podStartSLOduration=1.59735211 podStartE2EDuration="6.79048869s" totalImagesPullingTime="5.19313658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:55.788998757 +0000 UTC m=+57.876248053" watchObservedRunningTime="2026-07-09 18:03:55.79048869 +0000 UTC m=+57.877737995"
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.739901    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/1.log"
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.740408    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/0.log"
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.740448    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab136634-580b-4ce5-80fa-6c52e0e7e747" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77" exitCode=255
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.741291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-89h9v" event={"ID":"ab136634-580b-4ce5-80fa-6c52e0e7e747","Type":"ContainerDied","Data":"78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"}
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.741389    2565 scope.go:117] "RemoveContainer" containerID="2b5c4c3bbef0d456c8de7ce922bc149201285232681809c428875280d61d685b"
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.741477    2565 scope.go:117] "RemoveContainer" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"
Jul 09 18:03:56.743535 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:56.741712    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-89h9v_openshift-console-operator(ab136634-580b-4ce5-80fa-6c52e0e7e747)\"" pod="openshift-console-operator/console-operator-7866497494-89h9v" podUID="ab136634-580b-4ce5-80fa-6c52e0e7e747"
Jul 09 18:03:56.775934 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.775842    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x4475"]
Jul 09 18:03:56.780780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.780738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.786284 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:56.786248    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 09 18:03:56.786434 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:56.786335    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-f5z8w\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5z8w\"" type="*v1.Secret"
Jul 09 18:03:56.786434 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:56.786388    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Jul 09 18:03:56.786809 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:56.786544    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-x4475\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="9d071fa4-9ca5-46d3-85cd-c4af35330ddf" pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.804617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.804591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4475"]
Jul 09 18:03:56.821526 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.821499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.821828 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.821799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.822479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.821991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-crio-socket\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.822479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.822159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-data-volume\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.822479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.822430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhfpw\" (UniqueName: \"kubernetes.io/projected/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-api-access-fhfpw\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfpw\" (UniqueName: \"kubernetes.io/projected/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-api-access-fhfpw\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923541 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-crio-socket\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923708 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-data-volume\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.923821 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.923798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-crio-socket\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.924116 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.924095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-data-volume\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:56.932847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:56.932820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfpw\" (UniqueName: \"kubernetes.io/projected/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-api-access-fhfpw\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:57.429142 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.429107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:03:57.429329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.429166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:03:57.429329 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.429298    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:03:57.429406 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.429326    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:57.429406 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.429374    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.429357046 +0000 UTC m=+67.516606331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:03:57.429406 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.429389    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.429382786 +0000 UTC m=+67.516632068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:03:57.530256 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.530201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:03:57.530428 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.530354    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:03:57.530428 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.530421    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.530405692 +0000 UTC m=+67.617654975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:03:57.590250 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.590197    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:03:57.594094 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.594072    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:03:57.594150 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.594144    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:03:58.094125709 +0000 UTC m=+60.181374994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : configmap "kube-rbac-proxy" not found
Jul 09 18:03:57.631666 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.631631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:57.631812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.631674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:03:57.631812 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.631784    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:03:57.631883 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.631788    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.63177049 +0000 UTC m=+67.719019772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:03:57.631924 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.631908    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.631895332 +0000 UTC m=+67.719144613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:03:57.744610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.744585    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/1.log"
Jul 09 18:03:57.744979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.744914    2565 scope.go:117] "RemoveContainer" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"
Jul 09 18:03:57.745076 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.745057    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-89h9v_openshift-console-operator(ab136634-580b-4ce5-80fa-6c52e0e7e747)\"" pod="openshift-console-operator/console-operator-7866497494-89h9v" podUID="ab136634-580b-4ce5-80fa-6c52e0e7e747"
Jul 09 18:03:57.764841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.764815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82jsv_6430784f-f90b-4d10-9461-d80c0d848b40/dns-node-resolver/0.log"
Jul 09 18:03:57.772596 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:57.772577    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5z8w\""
Jul 09 18:03:57.924227 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.924184    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Jul 09 18:03:57.924414 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:57.924284    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:03:58.424266065 +0000 UTC m=+60.511515348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : failed to sync secret cache: timed out waiting for the condition
Jul 09 18:03:58.124893 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.124817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:03:58.136540 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.136508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:58.136676 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:58.136626    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:03:58.136714 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:58.136687    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:03:59.136671156 +0000 UTC m=+61.223920438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : configmap "kube-rbac-proxy" not found
Jul 09 18:03:58.439500 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.439465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:58.439650 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:58.439600    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:58.439694 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:58.439661    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:03:59.439644758 +0000 UTC m=+61.526894040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:58.495530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.495495    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"]
Jul 09 18:03:58.527243 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.527207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"]
Jul 09 18:03:58.527399 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.527328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"
Jul 09 18:03:58.529789 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.529770    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 18:03:58.529923 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.529805    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:58.530058 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.530045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hm89b\""
Jul 09 18:03:58.641033 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.640992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwr9d\" (UniqueName: \"kubernetes.io/projected/3ae4b84a-688a-42ff-9feb-3463e8cc35d5-kube-api-access-xwr9d\") pod \"migrator-7f59b54f6c-qx52v\" (UID: \"3ae4b84a-688a-42ff-9feb-3463e8cc35d5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"
Jul 09 18:03:58.742669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.742579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwr9d\" (UniqueName: \"kubernetes.io/projected/3ae4b84a-688a-42ff-9feb-3463e8cc35d5-kube-api-access-xwr9d\") pod \"migrator-7f59b54f6c-qx52v\" (UID: \"3ae4b84a-688a-42ff-9feb-3463e8cc35d5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"
Jul 09 18:03:58.751754 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.751727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwr9d\" (UniqueName: \"kubernetes.io/projected/3ae4b84a-688a-42ff-9feb-3463e8cc35d5-kube-api-access-xwr9d\") pod \"migrator-7f59b54f6c-qx52v\" (UID: \"3ae4b84a-688a-42ff-9feb-3463e8cc35d5\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"
Jul 09 18:03:58.835532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.835498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"
Jul 09 18:03:58.969559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.969532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdz76_44981ba1-e2ac-4bca-83c9-6988a5237880/node-ca/0.log"
Jul 09 18:03:58.987718 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:58.987568    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v"]
Jul 09 18:03:58.990120 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:03:58.990092    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae4b84a_688a_42ff_9feb_3463e8cc35d5.slice/crio-31f615ef26112fff053ce666b7ff8ab2d82b9f8a4b0defe2a97224dfcac8e009 WatchSource:0}: Error finding container 31f615ef26112fff053ce666b7ff8ab2d82b9f8a4b0defe2a97224dfcac8e009: Status 404 returned error can't find the container with id 31f615ef26112fff053ce666b7ff8ab2d82b9f8a4b0defe2a97224dfcac8e009
Jul 09 18:03:59.147393 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.147360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:59.147559 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:59.147498    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:03:59.147595 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:59.147571    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.147554077 +0000 UTC m=+63.234803359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : configmap "kube-rbac-proxy" not found
Jul 09 18:03:59.449920 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.449885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:03:59.450064 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:59.450036    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:59.450109 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:59.450100    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.450083788 +0000 UTC m=+63.537333069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:59.565459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.565432    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgpt5_0af21ee9-3d83-4aa9-a541-ddb07176e777/serve-healthcheck-canary/0.log"
Jul 09 18:03:59.755842 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.755710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v" event={"ID":"3ae4b84a-688a-42ff-9feb-3463e8cc35d5","Type":"ContainerStarted","Data":"31f615ef26112fff053ce666b7ff8ab2d82b9f8a4b0defe2a97224dfcac8e009"}
Jul 09 18:03:59.905957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.905918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:59.906124 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.905991    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:03:59.906454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:03:59.906436    2565 scope.go:117] "RemoveContainer" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"
Jul 09 18:03:59.906643 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:03:59.906621    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-89h9v_openshift-console-operator(ab136634-580b-4ce5-80fa-6c52e0e7e747)\"" pod="openshift-console-operator/console-operator-7866497494-89h9v" podUID="ab136634-580b-4ce5-80fa-6c52e0e7e747"
Jul 09 18:04:00.760333 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:00.760295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v" event={"ID":"3ae4b84a-688a-42ff-9feb-3463e8cc35d5","Type":"ContainerStarted","Data":"04a4d53960541ec38809140b83c4341b73575e45b53809a9a5ed7d09a8bef9ae"}
Jul 09 18:04:00.760708 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:00.760351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v" event={"ID":"3ae4b84a-688a-42ff-9feb-3463e8cc35d5","Type":"ContainerStarted","Data":"038e632f6c9af0e5710863946639162c6f91f8a010843956495b98693f1079bf"}
Jul 09 18:04:00.779581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:00.779533    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-qx52v" podStartSLOduration=1.335709711 podStartE2EDuration="2.779517274s" totalImagesPullingTime="1.443807563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:00.777609557 +0000 UTC m=+62.864858861" watchObservedRunningTime="2026-07-09 18:04:00.779517274 +0000 UTC m=+62.866766577"
Jul 09 18:04:01.167595 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:01.167494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:01.167757 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:01.167619    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:01.167757 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:01.167685    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.167669552 +0000 UTC m=+67.254918834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:01.469581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:01.469537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:01.469771 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:01.469687    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:01.469771 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:01.469752    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.469736614 +0000 UTC m=+67.556985897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:03.647504 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:03.647477    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqxhn"
Jul 09 18:04:04.193364 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.193317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:04:04.196029 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.196002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ql4\" (UniqueName: \"kubernetes.io/projected/35f8a758-2b3d-4c30-9ba0-f946a933e4f1-kube-api-access-g6ql4\") pod \"network-check-target-njgpn\" (UID: \"35f8a758-2b3d-4c30-9ba0-f946a933e4f1\") " pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:04:04.294533 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.294499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:04:04.294720 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.294624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:04:04.297441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.297417    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:04.297564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.297534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:04.305535 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:04.305516    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:04:04.305591 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:04.305585    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs podName:8d46ab2f-e328-4fc2-9c11-9ef297a01f63 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:08.305563942 +0000 UTC m=+130.392813241 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs") pod "network-metrics-daemon-hjjkk" (UID: "8d46ab2f-e328-4fc2-9c11-9ef297a01f63") : secret "metrics-daemon-secret" not found
Jul 09 18:04:04.307658 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.307638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b38b34-371d-43da-9ed4-6da5eb6ee973-original-pull-secret\") pod \"global-pull-secret-syncer-kmbmg\" (UID: \"24b38b34-371d-43da-9ed4-6da5eb6ee973\") " pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:04:04.391154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.391114    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmbmg"
Jul 09 18:04:04.489756 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.489725    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hzqhq\""
Jul 09 18:04:04.497286 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.497261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:04:04.516053 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.516025    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmbmg"]
Jul 09 18:04:04.518970 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:04.518943    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b38b34_371d_43da_9ed4_6da5eb6ee973.slice/crio-6a74a0719f6dcbd5211290dc29307de3b811dbc19e15598e8e2472646e2da0e5 WatchSource:0}: Error finding container 6a74a0719f6dcbd5211290dc29307de3b811dbc19e15598e8e2472646e2da0e5: Status 404 returned error can't find the container with id 6a74a0719f6dcbd5211290dc29307de3b811dbc19e15598e8e2472646e2da0e5
Jul 09 18:04:04.630038 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.630007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njgpn"]
Jul 09 18:04:04.633670 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:04.633642    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f8a758_2b3d_4c30_9ba0_f946a933e4f1.slice/crio-97002bf67839ab4199e1d294c72ad186cc1370dc5ffd4948d208e66f46278e33 WatchSource:0}: Error finding container 97002bf67839ab4199e1d294c72ad186cc1370dc5ffd4948d208e66f46278e33: Status 404 returned error can't find the container with id 97002bf67839ab4199e1d294c72ad186cc1370dc5ffd4948d208e66f46278e33
Jul 09 18:04:04.773024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.772940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njgpn" event={"ID":"35f8a758-2b3d-4c30-9ba0-f946a933e4f1","Type":"ContainerStarted","Data":"e4dcaf66d26788024c10cd4f8ab6c0c9763b86536b9501c485dd0e7966691d90"}
Jul 09 18:04:04.773024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.772988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njgpn" event={"ID":"35f8a758-2b3d-4c30-9ba0-f946a933e4f1","Type":"ContainerStarted","Data":"97002bf67839ab4199e1d294c72ad186cc1370dc5ffd4948d208e66f46278e33"}
Jul 09 18:04:04.773492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.773044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:04:04.774126 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.774104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kmbmg" event={"ID":"24b38b34-371d-43da-9ed4-6da5eb6ee973","Type":"ContainerStarted","Data":"6a74a0719f6dcbd5211290dc29307de3b811dbc19e15598e8e2472646e2da0e5"}
Jul 09 18:04:04.793825 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:04.793779    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njgpn" podStartSLOduration=65.793766216 podStartE2EDuration="1m5.793766216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:04.792170363 +0000 UTC m=+66.879419668" watchObservedRunningTime="2026-07-09 18:04:04.793766216 +0000 UTC m=+66.881015498"
Jul 09 18:04:05.201513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.201475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:05.201668 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.201573    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:05.201668 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.201656    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:13.201633776 +0000 UTC m=+75.288883079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:05.503698 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.503610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:05.503698 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.503664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.503715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503786    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503850    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503882    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503862    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:13.503839853 +0000 UTC m=+75.591089139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:05.503923 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503924    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls podName:bc24e4ed-e41e-4cd5-b979-1760c229d004 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.503906608 +0000 UTC m=+83.591155905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls") pod "cluster-samples-operator-65db858894-r95tx" (UID: "bc24e4ed-e41e-4cd5-b979-1760c229d004") : secret "samples-operator-tls" not found
Jul 09 18:04:05.504138 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.503948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls podName:5073d86a-582f-4f04-ad5a-130ea337c439 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.503935717 +0000 UTC m=+83.591185011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-v7s7l" (UID: "5073d86a-582f-4f04-ad5a-130ea337c439") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:04:05.605005 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.604967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:04:05.605174 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.605122    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:04:05.605222 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.605188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert podName:e9a8c762-322e-4594-b759-311e8ed6b8ee nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.605172126 +0000 UTC m=+83.692421409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-zml89" (UID: "e9a8c762-322e-4594-b759-311e8ed6b8ee") : secret "networking-console-plugin-cert" not found
Jul 09 18:04:05.706268 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.706219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:05.706448 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:05.706288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:05.706448 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.706353    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:04:05.706554 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.706462    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.706438822 +0000 UTC m=+83.793688107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : configmap references non-existent config key: service-ca.crt
Jul 09 18:04:05.706554 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:05.706494    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs podName:bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.706481546 +0000 UTC m=+83.793730835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs") pod "router-default-6fd9c96588-sbl55" (UID: "bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0") : secret "router-metrics-certs-default" not found
Jul 09 18:04:09.793108 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:09.793068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kmbmg" event={"ID":"24b38b34-371d-43da-9ed4-6da5eb6ee973","Type":"ContainerStarted","Data":"b28400115ac9ea240072e2d02ac39f8d1b7340afe65e90d8422da0e801a117a0"}
Jul 09 18:04:09.809657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:09.809606    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kmbmg" podStartSLOduration=66.519717606 podStartE2EDuration="1m10.809587458s" totalImagesPullingTime="4.289869852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:09.808592577 +0000 UTC m=+71.895841881" watchObservedRunningTime="2026-07-09 18:04:09.809587458 +0000 UTC m=+71.896836762"
Jul 09 18:04:10.660524 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.660479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:04:10.660767 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.660730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:04:10.662915 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.662882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d432f1b-b963-4d1a-bec9-052434b9454d-metrics-tls\") pod \"dns-default-zt2s9\" (UID: \"1d432f1b-b963-4d1a-bec9-052434b9454d\") " pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:04:10.663051 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.662955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"image-registry-599847f6ff-wx5zb\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") " pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:04:10.794391 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.794365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pxfrt\""
Jul 09 18:04:10.802371 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.802347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:04:10.808825 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.808767    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ln8j9\""
Jul 09 18:04:10.817102 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.817081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:04:10.978552 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.978522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:04:10.982752 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:10.982728    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f76cc9_2a12_46be_b7e8_fbb3258112a6.slice/crio-776e243d5c6f91e2dfe20c1d5e7093934d51ab2226447691c5683b05445530a4 WatchSource:0}: Error finding container 776e243d5c6f91e2dfe20c1d5e7093934d51ab2226447691c5683b05445530a4: Status 404 returned error can't find the container with id 776e243d5c6f91e2dfe20c1d5e7093934d51ab2226447691c5683b05445530a4
Jul 09 18:04:10.990720 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:10.990688    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt2s9"]
Jul 09 18:04:10.994760 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:10.994734    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d432f1b_b963_4d1a_bec9_052434b9454d.slice/crio-f3c15b194f2cf0523ebbe6ed4802fd0e559fafb2c78346c7b39b3a6e210b6925 WatchSource:0}: Error finding container f3c15b194f2cf0523ebbe6ed4802fd0e559fafb2c78346c7b39b3a6e210b6925: Status 404 returned error can't find the container with id f3c15b194f2cf0523ebbe6ed4802fd0e559fafb2c78346c7b39b3a6e210b6925
Jul 09 18:04:11.801960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:11.801921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt2s9" event={"ID":"1d432f1b-b963-4d1a-bec9-052434b9454d","Type":"ContainerStarted","Data":"f3c15b194f2cf0523ebbe6ed4802fd0e559fafb2c78346c7b39b3a6e210b6925"}
Jul 09 18:04:11.803391 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:11.803366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" event={"ID":"76f76cc9-2a12-46be-b7e8-fbb3258112a6","Type":"ContainerStarted","Data":"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"}
Jul 09 18:04:11.803529 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:11.803405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" event={"ID":"76f76cc9-2a12-46be-b7e8-fbb3258112a6","Type":"ContainerStarted","Data":"776e243d5c6f91e2dfe20c1d5e7093934d51ab2226447691c5683b05445530a4"}
Jul 09 18:04:11.803529 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:11.803506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:04:11.824442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:11.824385    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" podStartSLOduration=72.824365947 podStartE2EDuration="1m12.824365947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:11.823428019 +0000 UTC m=+73.910677322" watchObservedRunningTime="2026-07-09 18:04:11.824365947 +0000 UTC m=+73.911615252"
Jul 09 18:04:13.298761 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.298722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:13.299324 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.299299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:13.476719 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.476694    2565 scope.go:117] "RemoveContainer" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"
Jul 09 18:04:13.601111 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.601087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:13.601262 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:13.601203    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:13.601376 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:13.601356    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls podName:9d071fa4-9ca5-46d3-85cd-c4af35330ddf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:29.601336144 +0000 UTC m=+91.688585441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4475" (UID: "9d071fa4-9ca5-46d3-85cd-c4af35330ddf") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:13.812470 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.812385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt2s9" event={"ID":"1d432f1b-b963-4d1a-bec9-052434b9454d","Type":"ContainerStarted","Data":"16567b7501028a72f881185b4039ebe28935dd222d8cff222ed3e7adbc2909f6"}
Jul 09 18:04:13.812470 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.812468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt2s9" event={"ID":"1d432f1b-b963-4d1a-bec9-052434b9454d","Type":"ContainerStarted","Data":"d9d2f88982f0d861b416c03e0d8dd186c21e967679ab0ee77c52b4f79e4c37a3"}
Jul 09 18:04:13.812697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.812507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:04:13.813873 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.813850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:04:13.814215 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.814202    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/1.log"
Jul 09 18:04:13.814304 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.814243    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab136634-580b-4ce5-80fa-6c52e0e7e747" containerID="14314d4f255097aa2e1ccc6a532ac97767143aa19ceff2e7439111ce400852d1" exitCode=255
Jul 09 18:04:13.814304 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.814266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-89h9v" event={"ID":"ab136634-580b-4ce5-80fa-6c52e0e7e747","Type":"ContainerDied","Data":"14314d4f255097aa2e1ccc6a532ac97767143aa19ceff2e7439111ce400852d1"}
Jul 09 18:04:13.814411 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.814321    2565 scope.go:117] "RemoveContainer" containerID="78e8b7258f8c48a3a5dfcf5108f1630aea94315cad777834cfbc2592276b4b77"
Jul 09 18:04:13.814572 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.814558    2565 scope.go:117] "RemoveContainer" containerID="14314d4f255097aa2e1ccc6a532ac97767143aa19ceff2e7439111ce400852d1"
Jul 09 18:04:13.814772 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:13.814749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-89h9v_openshift-console-operator(ab136634-580b-4ce5-80fa-6c52e0e7e747)\"" pod="openshift-console-operator/console-operator-7866497494-89h9v" podUID="ab136634-580b-4ce5-80fa-6c52e0e7e747"
Jul 09 18:04:13.831961 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:13.831923    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zt2s9" podStartSLOduration=33.364876066 podStartE2EDuration="35.831910566s" totalImagesPullingTime="2.4670345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:13.830896066 +0000 UTC m=+75.918145382" watchObservedRunningTime="2026-07-09 18:04:13.831910566 +0000 UTC m=+75.919159869"
Jul 09 18:04:14.819431 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:14.819402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:04:19.905252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:19.905196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:04:19.905641 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:19.905274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:04:19.905641 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:19.905604    2565 scope.go:117] "RemoveContainer" containerID="14314d4f255097aa2e1ccc6a532ac97767143aa19ceff2e7439111ce400852d1"
Jul 09 18:04:19.905799 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:19.905778    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-89h9v_openshift-console-operator(ab136634-580b-4ce5-80fa-6c52e0e7e747)\"" pod="openshift-console-operator/console-operator-7866497494-89h9v" podUID="ab136634-580b-4ce5-80fa-6c52e0e7e747"
Jul 09 18:04:21.563904 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.563782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:04:21.563904 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.563830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:04:21.566508 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.566482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc24e4ed-e41e-4cd5-b979-1760c229d004-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-r95tx\" (UID: \"bc24e4ed-e41e-4cd5-b979-1760c229d004\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:04:21.566579 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.566482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5073d86a-582f-4f04-ad5a-130ea337c439-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-v7s7l\" (UID: \"5073d86a-582f-4f04-ad5a-130ea337c439\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:04:21.665320 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.665290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:04:21.667574 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.667553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a8c762-322e-4594-b759-311e8ed6b8ee-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-zml89\" (UID: \"e9a8c762-322e-4594-b759-311e8ed6b8ee\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:04:21.766046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.766009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:21.766266 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.766055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:21.766701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.766668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-service-ca-bundle\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:21.768475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.768451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0-metrics-certs\") pod \"router-default-6fd9c96588-sbl55\" (UID: \"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0\") " pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:21.820132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.820057    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-24rmh\""
Jul 09 18:04:21.828015 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.827995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"
Jul 09 18:04:21.833109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.833092    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qr8j7\""
Jul 09 18:04:21.841280 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.841258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"
Jul 09 18:04:21.926089 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.926064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f92d9\""
Jul 09 18:04:21.934128 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.934105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89"
Jul 09 18:04:21.962149 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.962118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4whfj\""
Jul 09 18:04:21.970587 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.970566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:21.973198 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.973174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx"]
Jul 09 18:04:21.987103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:21.987084    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l"]
Jul 09 18:04:22.088464 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.088432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-zml89"]
Jul 09 18:04:22.091010 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:22.090965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a8c762_322e_4594_b759_311e8ed6b8ee.slice/crio-972aad7c83c38a85cbfdf5e8e145e12c4cb180349c0bc3c8c41ca7a83e1dca39 WatchSource:0}: Error finding container 972aad7c83c38a85cbfdf5e8e145e12c4cb180349c0bc3c8c41ca7a83e1dca39: Status 404 returned error can't find the container with id 972aad7c83c38a85cbfdf5e8e145e12c4cb180349c0bc3c8c41ca7a83e1dca39
Jul 09 18:04:22.126341 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.126193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fd9c96588-sbl55"]
Jul 09 18:04:22.128806 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:22.128777    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3fdf67_d3ba_4cb1_90ef_0ea9498ae4e0.slice/crio-45a05bffcefa8cff4104d8a9b140125c15765e90be4dd42b636cbee87e988f6f WatchSource:0}: Error finding container 45a05bffcefa8cff4104d8a9b140125c15765e90be4dd42b636cbee87e988f6f: Status 404 returned error can't find the container with id 45a05bffcefa8cff4104d8a9b140125c15765e90be4dd42b636cbee87e988f6f
Jul 09 18:04:22.849389 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.849353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fd9c96588-sbl55" event={"ID":"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0","Type":"ContainerStarted","Data":"6ae62abd21f92867ac49a2e6107386c91af1c61f590e537dfa2ce59cf8465ff2"}
Jul 09 18:04:22.849867 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.849417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fd9c96588-sbl55" event={"ID":"bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0","Type":"ContainerStarted","Data":"45a05bffcefa8cff4104d8a9b140125c15765e90be4dd42b636cbee87e988f6f"}
Jul 09 18:04:22.850753 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.850728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l" event={"ID":"5073d86a-582f-4f04-ad5a-130ea337c439","Type":"ContainerStarted","Data":"996ce4eaad3acfccc45faa560321e271c94b1276b3e0b078046d0dee89f1ccc4"}
Jul 09 18:04:22.851964 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.851931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx" event={"ID":"bc24e4ed-e41e-4cd5-b979-1760c229d004","Type":"ContainerStarted","Data":"04d5d81a44863c9fd31eb6943a08811153dd74fa1bcba5e03ddd6569eaa10e10"}
Jul 09 18:04:22.858396 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.858366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89" event={"ID":"e9a8c762-322e-4594-b759-311e8ed6b8ee","Type":"ContainerStarted","Data":"972aad7c83c38a85cbfdf5e8e145e12c4cb180349c0bc3c8c41ca7a83e1dca39"}
Jul 09 18:04:22.971569 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.971527    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:22.974364 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.974342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:22.999404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:22.999172    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6fd9c96588-sbl55" podStartSLOduration=33.999149785 podStartE2EDuration="33.999149785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:22.880055701 +0000 UTC m=+84.967305004" watchObservedRunningTime="2026-07-09 18:04:22.999149785 +0000 UTC m=+85.086399090"
Jul 09 18:04:23.822738 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:23.822702    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zt2s9"
Jul 09 18:04:23.863614 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:23.863577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:23.865219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:23.865194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6fd9c96588-sbl55"
Jul 09 18:04:25.873046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.873007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89" event={"ID":"e9a8c762-322e-4594-b759-311e8ed6b8ee","Type":"ContainerStarted","Data":"11d0b891e2c30cafc801946313f010bbe16d2bcd84999f0192c14f218ae2be4f"}
Jul 09 18:04:25.874514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.874489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l" event={"ID":"5073d86a-582f-4f04-ad5a-130ea337c439","Type":"ContainerStarted","Data":"1e908e43457548b36b3be8ef761f5b917fac8f87b419d9763c77ea7fed1b71ea"}
Jul 09 18:04:25.876265 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.876221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx" event={"ID":"bc24e4ed-e41e-4cd5-b979-1760c229d004","Type":"ContainerStarted","Data":"483b1724f64e83ce8e692aafab9cc302fffe1c7d81a5acb0fab6ac14c986933a"}
Jul 09 18:04:25.876354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.876290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx" event={"ID":"bc24e4ed-e41e-4cd5-b979-1760c229d004","Type":"ContainerStarted","Data":"c604e523ae86dc35514bbddbb3b63473e29e3370a3410af664806c8507d9b150"}
Jul 09 18:04:25.895709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.895658    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-zml89" podStartSLOduration=33.814809524 podStartE2EDuration="36.895640615s" totalImagesPullingTime="3.080831091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:25.895511286 +0000 UTC m=+87.982760590" watchObservedRunningTime="2026-07-09 18:04:25.895640615 +0000 UTC m=+87.982889921"
Jul 09 18:04:25.932967 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:25.932918    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-r95tx" podStartSLOduration=33.784262914 podStartE2EDuration="36.932901962s" totalImagesPullingTime="3.148639048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:25.931389084 +0000 UTC m=+88.018638389" watchObservedRunningTime="2026-07-09 18:04:25.932901962 +0000 UTC m=+88.020151269"
Jul 09 18:04:29.628040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.627992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:29.630338 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.630312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d071fa4-9ca5-46d3-85cd-c4af35330ddf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4475\" (UID: \"9d071fa4-9ca5-46d3-85cd-c4af35330ddf\") " pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:29.795136 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.795106    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5z8w\""
Jul 09 18:04:29.803076 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.803056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4475"
Jul 09 18:04:29.930525 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.930467    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l" podStartSLOduration=37.742928539 podStartE2EDuration="40.930448848s" totalImagesPullingTime="3.187520309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:25.977183489 +0000 UTC m=+88.064432794" watchObservedRunningTime="2026-07-09 18:04:29.930448848 +0000 UTC m=+92.017698160"
Jul 09 18:04:29.931054 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:29.931038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4475"]
Jul 09 18:04:29.933496 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:29.933460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d071fa4_9ca5_46d3_85cd_c4af35330ddf.slice/crio-50ec019553cada7575ee3d404fcfa398607a8145683c3e84904f6d411c843b9d WatchSource:0}: Error finding container 50ec019553cada7575ee3d404fcfa398607a8145683c3e84904f6d411c843b9d: Status 404 returned error can't find the container with id 50ec019553cada7575ee3d404fcfa398607a8145683c3e84904f6d411c843b9d
Jul 09 18:04:30.810255 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:30.810125    2565 patch_prober.go:28] interesting pod/image-registry-599847f6ff-wx5zb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:04:30.810255 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:30.810188    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:04:30.897941 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:30.897842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4475" event={"ID":"9d071fa4-9ca5-46d3-85cd-c4af35330ddf","Type":"ContainerStarted","Data":"50ec019553cada7575ee3d404fcfa398607a8145683c3e84904f6d411c843b9d"}
Jul 09 18:04:32.811688 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:32.811657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:04:34.476570 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.476539    2565 scope.go:117] "RemoveContainer" containerID="14314d4f255097aa2e1ccc6a532ac97767143aa19ceff2e7439111ce400852d1"
Jul 09 18:04:34.915690 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.915612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4475" event={"ID":"9d071fa4-9ca5-46d3-85cd-c4af35330ddf","Type":"ContainerStarted","Data":"9b0b48df86eab98a3b6e2c905ccc35d92a162b2bebb13a30b8d5b4ac9f889a6b"}
Jul 09 18:04:34.915690 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.915683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4475" event={"ID":"9d071fa4-9ca5-46d3-85cd-c4af35330ddf","Type":"ContainerStarted","Data":"be9df42472784d665eb97aa7a7956c95dd321ef8000fb34e086d1ea8ee033b91"}
Jul 09 18:04:34.917412 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.917393    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:04:34.917528 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.917433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-89h9v" event={"ID":"ab136634-580b-4ce5-80fa-6c52e0e7e747","Type":"ContainerStarted","Data":"475f0c24bc981e6ceccc2a88c2ea2bc1f60a779d5af3f0bbd740c01bec413c8e"}
Jul 09 18:04:34.917799 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.917777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:04:34.935833 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:34.935765    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-89h9v" podStartSLOduration=40.499622228 podStartE2EDuration="45.935563112s" totalImagesPullingTime="5.435940884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:34.935309219 +0000 UTC m=+97.022558524" watchObservedRunningTime="2026-07-09 18:04:34.935563112 +0000 UTC m=+97.022812417"
Jul 09 18:04:35.086657 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:35.086626    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-89h9v"
Jul 09 18:04:35.780580 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:35.780549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njgpn"
Jul 09 18:04:36.469707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.469670    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jtb6d"]
Jul 09 18:04:36.478387 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.478357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.485252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.484982    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztzgw\""
Jul 09 18:04:36.485252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.485034    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:04:36.485878 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.485852    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:04:36.486002 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.485906    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:04:36.486190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.486170    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:04:36.587580 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587762 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587821 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-root\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587870 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-textfile\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-wtmp\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587984 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-metrics-client-ca\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.587984 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.587979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-sys\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.588077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.588001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.588077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.588041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4j6\" (UniqueName: \"kubernetes.io/projected/5c171010-2af7-4c10-8d31-f785c24d7900-kube-api-access-gj4j6\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.688841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.688790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4j6\" (UniqueName: \"kubernetes.io/projected/5c171010-2af7-4c10-8d31-f785c24d7900-kube-api-access-gj4j6\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689036 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.688865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689036 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.688929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689036 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.688950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-root\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689036 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.688972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-textfile\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-root\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-wtmp\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-metrics-client-ca\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-sys\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-sys\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-wtmp\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689731 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:36.689377    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 18:04:36.689731 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-textfile\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689731 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:04:36.689433    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls podName:5c171010-2af7-4c10-8d31-f785c24d7900 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:37.189414121 +0000 UTC m=+99.276663421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls") pod "node-exporter-jtb6d" (UID: "5c171010-2af7-4c10-8d31-f785c24d7900") : secret "node-exporter-tls" not found
Jul 09 18:04:36.689731 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.689731 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.689715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c171010-2af7-4c10-8d31-f785c24d7900-metrics-client-ca\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.691699 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.691672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:36.700621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:36.700541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4j6\" (UniqueName: \"kubernetes.io/projected/5c171010-2af7-4c10-8d31-f785c24d7900-kube-api-access-gj4j6\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:37.193153 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.193120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:37.195557 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.195535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c171010-2af7-4c10-8d31-f785c24d7900-node-exporter-tls\") pod \"node-exporter-jtb6d\" (UID: \"5c171010-2af7-4c10-8d31-f785c24d7900\") " pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:37.394520 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.394424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jtb6d"
Jul 09 18:04:37.405827 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:37.405801    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c171010_2af7_4c10_8d31_f785c24d7900.slice/crio-f58607d5b5502371b62bba1fe6c000b9416edb3b121c11b14ec391be79fdab5d WatchSource:0}: Error finding container f58607d5b5502371b62bba1fe6c000b9416edb3b121c11b14ec391be79fdab5d: Status 404 returned error can't find the container with id f58607d5b5502371b62bba1fe6c000b9416edb3b121c11b14ec391be79fdab5d
Jul 09 18:04:37.931767 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.931725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4475" event={"ID":"9d071fa4-9ca5-46d3-85cd-c4af35330ddf","Type":"ContainerStarted","Data":"3375267c4ce160c145dbf59d06e3998fd2e61989898ec383ae00dfdbd82b8593"}
Jul 09 18:04:37.933028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.932999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtb6d" event={"ID":"5c171010-2af7-4c10-8d31-f785c24d7900","Type":"ContainerStarted","Data":"f58607d5b5502371b62bba1fe6c000b9416edb3b121c11b14ec391be79fdab5d"}
Jul 09 18:04:37.970060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:37.970008    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x4475" podStartSLOduration=34.992739509 podStartE2EDuration="41.969992653s" totalImagesPullingTime="6.977253144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.966885568 +0000 UTC m=+100.054134872" watchObservedRunningTime="2026-07-09 18:04:37.969992653 +0000 UTC m=+100.057241957"
Jul 09 18:04:38.939534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:38.939493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtb6d" event={"ID":"5c171010-2af7-4c10-8d31-f785c24d7900","Type":"ContainerStarted","Data":"02bc90a9691b56219fb9dd2915afba3a3f029937ffd76bd48ebc08eb9f66df70"}
Jul 09 18:04:39.944404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:39.944368    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c171010-2af7-4c10-8d31-f785c24d7900" containerID="02bc90a9691b56219fb9dd2915afba3a3f029937ffd76bd48ebc08eb9f66df70" exitCode=0
Jul 09 18:04:39.944801 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:39.944455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtb6d" event={"ID":"5c171010-2af7-4c10-8d31-f785c24d7900","Type":"ContainerDied","Data":"02bc90a9691b56219fb9dd2915afba3a3f029937ffd76bd48ebc08eb9f66df70"}
Jul 09 18:04:40.950847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:40.950812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtb6d" event={"ID":"5c171010-2af7-4c10-8d31-f785c24d7900","Type":"ContainerStarted","Data":"a5c2987720e81161b0f3bd85484a43fff8d312e3041753f3986af8cea5d5bc3d"}
Jul 09 18:04:40.951261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:40.950866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtb6d" event={"ID":"5c171010-2af7-4c10-8d31-f785c24d7900","Type":"ContainerStarted","Data":"adac024692783bc52c6944285f00d3e1d82c8d11dc402b78051691fc5dc3f1a1"}
Jul 09 18:04:41.028073 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:41.028020    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jtb6d" podStartSLOduration=3.589509189 podStartE2EDuration="5.027999328s" totalImagesPullingTime="1.346991091s" totalInitContainerRuntime="91.499048ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:41.018015703 +0000 UTC m=+103.105265008" watchObservedRunningTime="2026-07-09 18:04:41.027999328 +0000 UTC m=+103.115248614"
Jul 09 18:04:45.329506 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.329472    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:04:45.596946 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.596866    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:04:45.601575 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.601555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.608563 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.608540    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:04:45.608841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.608814    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:04:45.608960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.608825    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:04:45.608960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.608909    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r7qp6\""
Jul 09 18:04:45.609069 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.609018    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:04:45.611609 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.611591    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:04:45.611768 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.611751    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:04:45.611856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.611797    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:04:45.632357 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.632331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:04:45.676840 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.676809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.676840 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.676839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.677016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.676861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.677016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.676934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.677016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.676958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.677016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.677000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rf5f\" (UniqueName: \"kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778246 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.778703 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.778506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rf5f\" (UniqueName: \"kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.779065 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.779036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.779186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.779156    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.779186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.779164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.780850 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.780825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.780979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.780964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.788994 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.788969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rf5f\" (UniqueName: \"kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f\") pod \"console-786878d9cc-fdslt\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") " pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:45.910701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:45.910617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:46.049677 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:46.049644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:04:46.052878 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:46.052848    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96afb73_d0d4_44e9_8f8c_616a367f7df2.slice/crio-ab37ccca90861321e30fc09115c38f0a5754822ca4c44bb3fcdf9c69e3f9c4c6 WatchSource:0}: Error finding container ab37ccca90861321e30fc09115c38f0a5754822ca4c44bb3fcdf9c69e3f9c4c6: Status 404 returned error can't find the container with id ab37ccca90861321e30fc09115c38f0a5754822ca4c44bb3fcdf9c69e3f9c4c6
Jul 09 18:04:46.975732 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:46.975689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786878d9cc-fdslt" event={"ID":"f96afb73-d0d4-44e9-8f8c-616a367f7df2","Type":"ContainerStarted","Data":"ab37ccca90861321e30fc09115c38f0a5754822ca4c44bb3fcdf9c69e3f9c4c6"}
Jul 09 18:04:49.988437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:49.988396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786878d9cc-fdslt" event={"ID":"f96afb73-d0d4-44e9-8f8c-616a367f7df2","Type":"ContainerStarted","Data":"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"}
Jul 09 18:04:50.008311 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:50.008263    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786878d9cc-fdslt" podStartSLOduration=1.7032852109999999 podStartE2EDuration="5.008224493s" totalImagesPullingTime="3.304939282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.006700447 +0000 UTC m=+112.093949751" watchObservedRunningTime="2026-07-09 18:04:50.008224493 +0000 UTC m=+112.095473795"
Jul 09 18:04:54.054931 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.054895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:04:54.058207 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.058189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.088757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.088718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:04:54.089835 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.089816    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:04:54.149989 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.149942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77m8\" (UniqueName: \"kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.149989 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.149994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.150212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.150042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.150212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.150062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.150212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.150121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.150212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.150162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.150212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.150205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251456 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77m8\" (UniqueName: \"kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251456 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251675 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251675 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251675 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251675 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.251881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.251733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.252280 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.252256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.252472 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.252301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.252472 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.252395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.252605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.252554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.254082 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.254058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.254159 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.254145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.266185 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.266165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77m8\" (UniqueName: \"kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8\") pod \"console-6cdbbf6644-v7fsr\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") " pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.366643 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.366565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:04:54.503934 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:54.503910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:04:54.506354 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:04:54.506328    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0627c69_a529_492e_bcbe_a6cd5314d408.slice/crio-4c21f0cb76b82f3794a3d599a2c73035c1a83219b1de6ad64d2da2297d4668a3 WatchSource:0}: Error finding container 4c21f0cb76b82f3794a3d599a2c73035c1a83219b1de6ad64d2da2297d4668a3: Status 404 returned error can't find the container with id 4c21f0cb76b82f3794a3d599a2c73035c1a83219b1de6ad64d2da2297d4668a3
Jul 09 18:04:55.007451 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.007413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbbf6644-v7fsr" event={"ID":"a0627c69-a529-492e-bcbe-a6cd5314d408","Type":"ContainerStarted","Data":"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"}
Jul 09 18:04:55.007618 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.007464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbbf6644-v7fsr" event={"ID":"a0627c69-a529-492e-bcbe-a6cd5314d408","Type":"ContainerStarted","Data":"4c21f0cb76b82f3794a3d599a2c73035c1a83219b1de6ad64d2da2297d4668a3"}
Jul 09 18:04:55.026017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.025966    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cdbbf6644-v7fsr" podStartSLOduration=1.025950056 podStartE2EDuration="1.025950056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:55.025332179 +0000 UTC m=+117.112581483" watchObservedRunningTime="2026-07-09 18:04:55.025950056 +0000 UTC m=+117.113199359"
Jul 09 18:04:55.910949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.910911    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:55.911374 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.910975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:55.915934 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:55.915909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:04:56.015571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:04:56.015547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:05:04.367188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:04.367154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:05:04.367679 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:04.367262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:05:04.372067 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:04.372044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:05:05.047636 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:05.047609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:05:05.125707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:05.125673    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:05:07.053089 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:07.053054    2565 generic.go:356] "Generic (PLEG): container finished" podID="2dc89993-a760-495b-9819-6307bcf51561" containerID="def76b1dd2f1f4fc691db9dc1b476393db66fd1a68dd98eea7dc6ddf6610045a" exitCode=0
Jul 09 18:05:07.053501 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:07.053134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc" event={"ID":"2dc89993-a760-495b-9819-6307bcf51561","Type":"ContainerDied","Data":"def76b1dd2f1f4fc691db9dc1b476393db66fd1a68dd98eea7dc6ddf6610045a"}
Jul 09 18:05:07.053581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:07.053566    2565 scope.go:117] "RemoveContainer" containerID="def76b1dd2f1f4fc691db9dc1b476393db66fd1a68dd98eea7dc6ddf6610045a"
Jul 09 18:05:08.058757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.058723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-jt9jc" event={"ID":"2dc89993-a760-495b-9819-6307bcf51561","Type":"ContainerStarted","Data":"4c53291be04f95b155aed3cf9c2c70b84874e3d52d68f45ef42fa0efc4343fdf"}
Jul 09 18:05:08.370260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.370149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:05:08.372663 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.372632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46ab2f-e328-4fc2-9c11-9ef297a01f63-metrics-certs\") pod \"network-metrics-daemon-hjjkk\" (UID: \"8d46ab2f-e328-4fc2-9c11-9ef297a01f63\") " pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:05:08.589789 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.589756    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmkm4\""
Jul 09 18:05:08.597353 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.597330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjjkk"
Jul 09 18:05:08.760979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:08.760948    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjjkk"]
Jul 09 18:05:08.768941 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:05:08.768913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d46ab2f_e328_4fc2_9c11_9ef297a01f63.slice/crio-67549a794bb74caafd93e416fd8b764b07cedfca9c1de1c1310195a72357c286 WatchSource:0}: Error finding container 67549a794bb74caafd93e416fd8b764b07cedfca9c1de1c1310195a72357c286: Status 404 returned error can't find the container with id 67549a794bb74caafd93e416fd8b764b07cedfca9c1de1c1310195a72357c286
Jul 09 18:05:09.063862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:09.063826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjkk" event={"ID":"8d46ab2f-e328-4fc2-9c11-9ef297a01f63","Type":"ContainerStarted","Data":"67549a794bb74caafd93e416fd8b764b07cedfca9c1de1c1310195a72357c286"}
Jul 09 18:05:10.350080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.349992    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerName="registry" containerID="cri-o://f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af" gracePeriod=30
Jul 09 18:05:10.598224 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.598200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:05:10.689732 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689705    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.689896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.689896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.689896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.689896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.689896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.690112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.690112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.689936    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8txt\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt\") pod \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\" (UID: \"76f76cc9-2a12-46be-b7e8-fbb3258112a6\") "
Jul 09 18:05:10.690214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.690157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:10.690296 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.690275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:10.692583 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.692556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt" (OuterVolumeSpecName: "kube-api-access-h8txt") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "kube-api-access-h8txt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:10.693198 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.693171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:10.693366 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.693219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:10.693366 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.693246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:10.693530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.693310    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:10.698342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.698319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "76f76cc9-2a12-46be-b7e8-fbb3258112a6" (UID: "76f76cc9-2a12-46be-b7e8-fbb3258112a6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:10.790910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790858    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-image-registry-private-configuration\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.790910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790902    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8txt\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-kube-api-access-h8txt\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.790910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790914    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-trusted-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.790910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790925    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-bound-sa-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.791171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790933    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-certificates\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.791171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790949    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76f76cc9-2a12-46be-b7e8-fbb3258112a6-registry-tls\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.791171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790957    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76f76cc9-2a12-46be-b7e8-fbb3258112a6-installation-pull-secrets\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:10.791171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:10.790965    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76f76cc9-2a12-46be-b7e8-fbb3258112a6-ca-trust-extracted\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:11.072582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.072487    2565 generic.go:356] "Generic (PLEG): container finished" podID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerID="f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af" exitCode=0
Jul 09 18:05:11.072582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.072540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb"
Jul 09 18:05:11.072811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.072572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" event={"ID":"76f76cc9-2a12-46be-b7e8-fbb3258112a6","Type":"ContainerDied","Data":"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"}
Jul 09 18:05:11.072811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.072631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-599847f6ff-wx5zb" event={"ID":"76f76cc9-2a12-46be-b7e8-fbb3258112a6","Type":"ContainerDied","Data":"776e243d5c6f91e2dfe20c1d5e7093934d51ab2226447691c5683b05445530a4"}
Jul 09 18:05:11.072811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.072662    2565 scope.go:117] "RemoveContainer" containerID="f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"
Jul 09 18:05:11.074518 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.074492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjkk" event={"ID":"8d46ab2f-e328-4fc2-9c11-9ef297a01f63","Type":"ContainerStarted","Data":"cbd0d4071ad664e8e10085f215576a2999b18eb31508d4cf9910cfaee3d65e8d"}
Jul 09 18:05:11.074624 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.074545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjjkk" event={"ID":"8d46ab2f-e328-4fc2-9c11-9ef297a01f63","Type":"ContainerStarted","Data":"eeb87536a5c7c97e6685ff938dc5767a8586b1bb151b89390189eeebc3e3204a"}
Jul 09 18:05:11.081270 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.081217    2565 scope.go:117] "RemoveContainer" containerID="f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"
Jul 09 18:05:11.081545 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:05:11.081526    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af\": container with ID starting with f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af not found: ID does not exist" containerID="f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"
Jul 09 18:05:11.081621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.081557    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af"} err="failed to get container status \"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af\": rpc error: code = NotFound desc = could not find container \"f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af\": container with ID starting with f11358f3ed51ca4f68a559ed67b50f265ddae643b820cef76f710d002c09a5af not found: ID does not exist"
Jul 09 18:05:11.140147 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.140091    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hjjkk" podStartSLOduration=130.83809518 podStartE2EDuration="2m12.140074459s" totalImagesPullingTime="1.301979279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:11.115092509 +0000 UTC m=+133.202341817" watchObservedRunningTime="2026-07-09 18:05:11.140074459 +0000 UTC m=+133.227323762"
Jul 09 18:05:11.141041 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.141024    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:05:11.144686 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:11.144664    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-599847f6ff-wx5zb"]
Jul 09 18:05:12.481746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:12.481708    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" path="/var/lib/kubelet/pods/76f76cc9-2a12-46be-b7e8-fbb3258112a6/volumes"
Jul 09 18:05:16.100472 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:16.100440    2565 generic.go:356] "Generic (PLEG): container finished" podID="44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3" containerID="b906ed95a1a63d31ae25a3bca1cb889de64489fe74570af45cff874a00eb5e08" exitCode=0
Jul 09 18:05:16.100837 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:16.100516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c" event={"ID":"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3","Type":"ContainerDied","Data":"b906ed95a1a63d31ae25a3bca1cb889de64489fe74570af45cff874a00eb5e08"}
Jul 09 18:05:16.100837 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:16.100831    2565 scope.go:117] "RemoveContainer" containerID="b906ed95a1a63d31ae25a3bca1cb889de64489fe74570af45cff874a00eb5e08"
Jul 09 18:05:16.812932 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:16.812894    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 18:05:16.833542 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:16.833506    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgpt5_0af21ee9-3d83-4aa9-a541-ddb07176e777/serve-healthcheck-canary/0.log"
Jul 09 18:05:17.105827 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:17.105748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x6k6c" event={"ID":"44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3","Type":"ContainerStarted","Data":"4f080574dd6e9fcc48218b2dca0200a675b5f78d60f7a1a6720dda361f93b2b2"}
Jul 09 18:05:22.129685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:22.129650    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5b51860-1dbb-4ff1-a7cb-0273f891f537" containerID="64221631b670122787015ad6d057ffa02a49668f54f7be961fd9a884b1badd37" exitCode=0
Jul 09 18:05:22.130043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:22.129700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx" event={"ID":"b5b51860-1dbb-4ff1-a7cb-0273f891f537","Type":"ContainerDied","Data":"64221631b670122787015ad6d057ffa02a49668f54f7be961fd9a884b1badd37"}
Jul 09 18:05:22.130043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:22.130004    2565 scope.go:117] "RemoveContainer" containerID="64221631b670122787015ad6d057ffa02a49668f54f7be961fd9a884b1badd37"
Jul 09 18:05:23.136751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:23.136714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-f5qgx" event={"ID":"b5b51860-1dbb-4ff1-a7cb-0273f891f537","Type":"ContainerStarted","Data":"ce9faba9b6485ab01d47ddbbd6c1515dacf4337ee276d24238934b7e10a8eef6"}
Jul 09 18:05:30.146840 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.146772    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-786878d9cc-fdslt" podUID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" containerName="console" containerID="cri-o://d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e" gracePeriod=15
Jul 09 18:05:30.388641 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.388617    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:05:30.449492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rf5f\" (UniqueName: \"kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449543    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config\") pod \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\" (UID: \"f96afb73-d0d4-44e9-8f8c-616a367f7df2\") "
Jul 09 18:05:30.449949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.449923    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca" (OuterVolumeSpecName: "service-ca") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:30.450119 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.450088    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config" (OuterVolumeSpecName: "console-config") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:30.450119 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.450093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:30.451874 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.451848    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:30.451951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.451869    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:30.451951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.451928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f" (OuterVolumeSpecName: "kube-api-access-9rf5f") pod "f96afb73-d0d4-44e9-8f8c-616a367f7df2" (UID: "f96afb73-d0d4-44e9-8f8c-616a367f7df2"). InnerVolumeSpecName "kube-api-access-9rf5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:30.550942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550898    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:30.550942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550938    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:30.550942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550954    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:30.551172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550967    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rf5f\" (UniqueName: \"kubernetes.io/projected/f96afb73-d0d4-44e9-8f8c-616a367f7df2-kube-api-access-9rf5f\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:30.551172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550979    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:30.551172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:30.550990    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f96afb73-d0d4-44e9-8f8c-616a367f7df2-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:05:31.169991 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.169952    2565 generic.go:356] "Generic (PLEG): container finished" podID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" containerID="d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e" exitCode=0
Jul 09 18:05:31.170457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.169997    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786878d9cc-fdslt"
Jul 09 18:05:31.170457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.170038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786878d9cc-fdslt" event={"ID":"f96afb73-d0d4-44e9-8f8c-616a367f7df2","Type":"ContainerDied","Data":"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"}
Jul 09 18:05:31.170457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.170091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786878d9cc-fdslt" event={"ID":"f96afb73-d0d4-44e9-8f8c-616a367f7df2","Type":"ContainerDied","Data":"ab37ccca90861321e30fc09115c38f0a5754822ca4c44bb3fcdf9c69e3f9c4c6"}
Jul 09 18:05:31.170457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.170116    2565 scope.go:117] "RemoveContainer" containerID="d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"
Jul 09 18:05:31.178113 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.178097    2565 scope.go:117] "RemoveContainer" containerID="d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"
Jul 09 18:05:31.178367 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:05:31.178349    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e\": container with ID starting with d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e not found: ID does not exist" containerID="d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"
Jul 09 18:05:31.178412 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.178375    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e"} err="failed to get container status \"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e\": rpc error: code = NotFound desc = could not find container \"d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e\": container with ID starting with d4ee1bfb50de5279cde2776a69eefad0a76b648504eefa549e1bb556f9b2167e not found: ID does not exist"
Jul 09 18:05:31.187425 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.187405    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:05:31.196501 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:31.196479    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786878d9cc-fdslt"]
Jul 09 18:05:32.481191 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:32.481159    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" path="/var/lib/kubelet/pods/f96afb73-d0d4-44e9-8f8c-616a367f7df2/volumes"
Jul 09 18:05:56.267055 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:56.267027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:05:56.267496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:56.267068    2565 generic.go:356] "Generic (PLEG): container finished" podID="5073d86a-582f-4f04-ad5a-130ea337c439" containerID="1e908e43457548b36b3be8ef761f5b917fac8f87b419d9763c77ea7fed1b71ea" exitCode=2
Jul 09 18:05:56.267496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:56.267144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l" event={"ID":"5073d86a-582f-4f04-ad5a-130ea337c439","Type":"ContainerDied","Data":"1e908e43457548b36b3be8ef761f5b917fac8f87b419d9763c77ea7fed1b71ea"}
Jul 09 18:05:56.267567 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:56.267515    2565 scope.go:117] "RemoveContainer" containerID="1e908e43457548b36b3be8ef761f5b917fac8f87b419d9763c77ea7fed1b71ea"
Jul 09 18:05:57.272435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:57.272410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:05:57.272861 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:05:57.272536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-v7s7l" event={"ID":"5073d86a-582f-4f04-ad5a-130ea337c439","Type":"ContainerStarted","Data":"fc790ea2643252d1dd461ddc8cb33573eb1ea4dad72b0ba74851da3db2ac1ff4"}
Jul 09 18:06:09.241448 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.241399    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:06:09.241978 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.241910    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerName="registry"
Jul 09 18:06:09.241978 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.241929    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerName="registry"
Jul 09 18:06:09.241978 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.241959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" containerName="console"
Jul 09 18:06:09.241978 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.241969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" containerName="console"
Jul 09 18:06:09.242164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.242045    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f76cc9-2a12-46be-b7e8-fbb3258112a6" containerName="registry"
Jul 09 18:06:09.242164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.242057    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96afb73-d0d4-44e9-8f8c-616a367f7df2" containerName="console"
Jul 09 18:06:09.245517 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.245488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.263121 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.263081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:06:09.266617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.266581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.266809 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.266639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.266809 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.266695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.267094 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.266835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.267094 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.266858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpd9\" (UniqueName: \"kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.267176 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.267117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.267176 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.267159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368034 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.367984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368034 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpd9\" (UniqueName: \"kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368977 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.368977 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.369126 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.368986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.369344 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.369325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.370807 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.370786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.370908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.370875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.378515 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.378488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpd9\" (UniqueName: \"kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9\") pod \"console-679f694db-5d22d\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") " pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.564316 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.564202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:09.704382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:09.704346    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:06:09.707599 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:06:09.707572    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2795553_e1f1_472e_abe9_9ff20923d15b.slice/crio-e279eb9292d8f8784678a83978c152a3309d88474a75481f88e9b3ed9ce72c1e WatchSource:0}: Error finding container e279eb9292d8f8784678a83978c152a3309d88474a75481f88e9b3ed9ce72c1e: Status 404 returned error can't find the container with id e279eb9292d8f8784678a83978c152a3309d88474a75481f88e9b3ed9ce72c1e
Jul 09 18:06:10.322997 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:10.322961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679f694db-5d22d" event={"ID":"d2795553-e1f1-472e-abe9-9ff20923d15b","Type":"ContainerStarted","Data":"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"}
Jul 09 18:06:10.323444 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:10.323018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679f694db-5d22d" event={"ID":"d2795553-e1f1-472e-abe9-9ff20923d15b","Type":"ContainerStarted","Data":"e279eb9292d8f8784678a83978c152a3309d88474a75481f88e9b3ed9ce72c1e"}
Jul 09 18:06:10.344254 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:10.344185    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679f694db-5d22d" podStartSLOduration=1.344168643 podStartE2EDuration="1.344168643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:10.342828665 +0000 UTC m=+192.430077968" watchObservedRunningTime="2026-07-09 18:06:10.344168643 +0000 UTC m=+192.431417947"
Jul 09 18:06:19.564846 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:19.564806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:19.565357 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:19.564882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:19.569917 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:19.569887    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:20.365197 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:20.365162    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:06:20.414459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:20.414420    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:06:45.438591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.438523    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cdbbf6644-v7fsr" podUID="a0627c69-a529-492e-bcbe-a6cd5314d408" containerName="console" containerID="cri-o://1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d" gracePeriod=15
Jul 09 18:06:45.693654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.693566    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:06:45.770409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q77m8\" (UniqueName: \"kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770613    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca\") pod \"a0627c69-a529-492e-bcbe-a6cd5314d408\" (UID: \"a0627c69-a529-492e-bcbe-a6cd5314d408\") "
Jul 09 18:06:45.770981 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:45.770981 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.770967    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config" (OuterVolumeSpecName: "console-config") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:45.771070 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.771002    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:45.771104 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.771062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca" (OuterVolumeSpecName: "service-ca") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:45.772897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.772866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8" (OuterVolumeSpecName: "kube-api-access-q77m8") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "kube-api-access-q77m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:45.773050 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.772969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:45.773050 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.772999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a0627c69-a529-492e-bcbe-a6cd5314d408" (UID: "a0627c69-a529-492e-bcbe-a6cd5314d408"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:45.871949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871889    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.871949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871937    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77m8\" (UniqueName: \"kubernetes.io/projected/a0627c69-a529-492e-bcbe-a6cd5314d408-kube-api-access-q77m8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.871949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871949    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-trusted-ca-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.871949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871957    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.871949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871966    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0627c69-a529-492e-bcbe-a6cd5314d408-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.872298 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871974    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:45.872298 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:45.871984    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0627c69-a529-492e-bcbe-a6cd5314d408-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.461064 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.461031    2565 generic.go:356] "Generic (PLEG): container finished" podID="a0627c69-a529-492e-bcbe-a6cd5314d408" containerID="1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d" exitCode=0
Jul 09 18:06:46.461516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.461081    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbbf6644-v7fsr"
Jul 09 18:06:46.461516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.461129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbbf6644-v7fsr" event={"ID":"a0627c69-a529-492e-bcbe-a6cd5314d408","Type":"ContainerDied","Data":"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"}
Jul 09 18:06:46.461516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.461178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbbf6644-v7fsr" event={"ID":"a0627c69-a529-492e-bcbe-a6cd5314d408","Type":"ContainerDied","Data":"4c21f0cb76b82f3794a3d599a2c73035c1a83219b1de6ad64d2da2297d4668a3"}
Jul 09 18:06:46.461516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.461200    2565 scope.go:117] "RemoveContainer" containerID="1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"
Jul 09 18:06:46.469815 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.469794    2565 scope.go:117] "RemoveContainer" containerID="1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"
Jul 09 18:06:46.470144 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:06:46.470123    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d\": container with ID starting with 1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d not found: ID does not exist" containerID="1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"
Jul 09 18:06:46.470198 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.470154    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d"} err="failed to get container status \"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d\": rpc error: code = NotFound desc = could not find container \"1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d\": container with ID starting with 1f7c96c92c64022aba8797621a2959f18f3af33a95cc26f712aac0cee9d9613d not found: ID does not exist"
Jul 09 18:06:46.482576 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.482537    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:06:46.488592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:46.488554    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdbbf6644-v7fsr"]
Jul 09 18:06:48.480897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:06:48.480862    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0627c69-a529-492e-bcbe-a6cd5314d408" path="/var/lib/kubelet/pods/a0627c69-a529-492e-bcbe-a6cd5314d408/volumes"
Jul 09 18:07:23.533368 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.533269    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:07:23.533772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.533646    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0627c69-a529-492e-bcbe-a6cd5314d408" containerName="console"
Jul 09 18:07:23.533772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.533663    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0627c69-a529-492e-bcbe-a6cd5314d408" containerName="console"
Jul 09 18:07:23.533772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.533711    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0627c69-a529-492e-bcbe-a6cd5314d408" containerName="console"
Jul 09 18:07:23.536698 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.536618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.547557 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.547532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:07:23.567653 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567653 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrb6\" (UniqueName: \"kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.567973 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.567858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrb6\" (UniqueName: \"kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.668809 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.668747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.669431 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.669404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.669563 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.669465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.669563 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.669460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.669563 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.669501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.670898 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.670867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.670988 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.670909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.677619 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.677600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrb6\" (UniqueName: \"kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6\") pod \"console-75b7685c4d-h4d4p\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") " pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.850126 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.850021    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:23.981886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:23.981859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:07:23.984325 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:07:23.984302    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6669d28_e3d0_459f_844a_8e0b8226b9d1.slice/crio-d31c30fadfb5771febb22a3242149047a884e908814f6020e15d654382ee701a WatchSource:0}: Error finding container d31c30fadfb5771febb22a3242149047a884e908814f6020e15d654382ee701a: Status 404 returned error can't find the container with id d31c30fadfb5771febb22a3242149047a884e908814f6020e15d654382ee701a
Jul 09 18:07:24.605310 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:24.605269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b7685c4d-h4d4p" event={"ID":"a6669d28-e3d0-459f-844a-8e0b8226b9d1","Type":"ContainerStarted","Data":"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"}
Jul 09 18:07:24.605801 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:24.605334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b7685c4d-h4d4p" event={"ID":"a6669d28-e3d0-459f-844a-8e0b8226b9d1","Type":"ContainerStarted","Data":"d31c30fadfb5771febb22a3242149047a884e908814f6020e15d654382ee701a"}
Jul 09 18:07:24.625469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:24.625415    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75b7685c4d-h4d4p" podStartSLOduration=1.625399824 podStartE2EDuration="1.625399824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:07:24.624353379 +0000 UTC m=+266.711602696" watchObservedRunningTime="2026-07-09 18:07:24.625399824 +0000 UTC m=+266.712649127"
Jul 09 18:07:33.850400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:33.850354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:33.850897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:33.850432    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:33.855119 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:33.855097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:34.648581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:34.648551    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:07:34.698746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:34.698715    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:07:58.341065 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.341032    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:07:58.345759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.345737    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:07:58.346593 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.346568    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:07:58.351163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.351141    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:07:58.352503 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.352485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:07:58.355645 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.355623    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:07:58.358803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:58.358776    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:07:59.726105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:59.726036    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679f694db-5d22d" podUID="d2795553-e1f1-472e-abe9-9ff20923d15b" containerName="console" containerID="cri-o://ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed" gracePeriod=15
Jul 09 18:07:59.971168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:07:59.971141    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:08:00.078900 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078822    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.078900 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqpd9\" (UniqueName: \"kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.078998    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config\") pod \"d2795553-e1f1-472e-abe9-9ff20923d15b\" (UID: \"d2795553-e1f1-472e-abe9-9ff20923d15b\") "
Jul 09 18:08:00.079418 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.079389    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:08:00.079507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.079399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:08:00.079507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.079452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:08:00.079507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.079464    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config" (OuterVolumeSpecName: "console-config") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:08:00.081385 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.081358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:08:00.081471 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.081407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:08:00.081471 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.081432    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9" (OuterVolumeSpecName: "kube-api-access-dqpd9") pod "d2795553-e1f1-472e-abe9-9ff20923d15b" (UID: "d2795553-e1f1-472e-abe9-9ff20923d15b"). InnerVolumeSpecName "kube-api-access-dqpd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:00.179590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179556    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-trusted-ca-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179584    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179596    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqpd9\" (UniqueName: \"kubernetes.io/projected/d2795553-e1f1-472e-abe9-9ff20923d15b-kube-api-access-dqpd9\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179604    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179613    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179621    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2795553-e1f1-472e-abe9-9ff20923d15b-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.179819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.179629    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2795553-e1f1-472e-abe9-9ff20923d15b-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:00.746129 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.746041    2565 generic.go:356] "Generic (PLEG): container finished" podID="d2795553-e1f1-472e-abe9-9ff20923d15b" containerID="ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed" exitCode=0
Jul 09 18:08:00.746129 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.746089    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679f694db-5d22d"
Jul 09 18:08:00.746566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.746128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679f694db-5d22d" event={"ID":"d2795553-e1f1-472e-abe9-9ff20923d15b","Type":"ContainerDied","Data":"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"}
Jul 09 18:08:00.746566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.746183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679f694db-5d22d" event={"ID":"d2795553-e1f1-472e-abe9-9ff20923d15b","Type":"ContainerDied","Data":"e279eb9292d8f8784678a83978c152a3309d88474a75481f88e9b3ed9ce72c1e"}
Jul 09 18:08:00.746566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.746210    2565 scope.go:117] "RemoveContainer" containerID="ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"
Jul 09 18:08:00.754517 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.754495    2565 scope.go:117] "RemoveContainer" containerID="ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"
Jul 09 18:08:00.754775 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:08:00.754753    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed\": container with ID starting with ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed not found: ID does not exist" containerID="ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"
Jul 09 18:08:00.754824 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.754783    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed"} err="failed to get container status \"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed\": rpc error: code = NotFound desc = could not find container \"ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed\": container with ID starting with ed52706419d6603461d754d5ec897bbb303a6f2154b0b797bc8d1291e0aa13ed not found: ID does not exist"
Jul 09 18:08:00.762261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.762216    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:08:00.764603 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:00.764581    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679f694db-5d22d"]
Jul 09 18:08:02.480484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:02.480451    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2795553-e1f1-472e-abe9-9ff20923d15b" path="/var/lib/kubelet/pods/d2795553-e1f1-472e-abe9-9ff20923d15b/volumes"
Jul 09 18:08:22.331009 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.330973    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"]
Jul 09 18:08:22.331546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.331266    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2795553-e1f1-472e-abe9-9ff20923d15b" containerName="console"
Jul 09 18:08:22.331546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.331282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2795553-e1f1-472e-abe9-9ff20923d15b" containerName="console"
Jul 09 18:08:22.331546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.331346    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2795553-e1f1-472e-abe9-9ff20923d15b" containerName="console"
Jul 09 18:08:22.334294 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.334277    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.337365 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.337341    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f4j7t\""
Jul 09 18:08:22.337736 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.337717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:08:22.338240 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.338219    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:08:22.348219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.348188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"]
Jul 09 18:08:22.436532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.436498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.436709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.436540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktgw\" (UniqueName: \"kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.436709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.436569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.537209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.537172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.537402 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.537221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktgw\" (UniqueName: \"kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.537402 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.537284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.537610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.537591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.537659 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.537639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.545909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.545885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktgw\" (UniqueName: \"kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.643085 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.642990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:22.775018 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.774986    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"]
Jul 09 18:08:22.778561 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:08:22.778533    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae10136c_d8d9_4fa0_b3b7_e5dc6e27cca2.slice/crio-5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff WatchSource:0}: Error finding container 5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff: Status 404 returned error can't find the container with id 5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff
Jul 09 18:08:22.780366 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.780349    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:08:22.830196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:22.830162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955" event={"ID":"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2","Type":"ContainerStarted","Data":"5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff"}
Jul 09 18:08:32.874053 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:32.874017    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerID="f197ee8f60a6cde41abb7f3399971fea2a37fd2be9d312527194a634d04b651d" exitCode=0
Jul 09 18:08:32.874450 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:32.874065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955" event={"ID":"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2","Type":"ContainerDied","Data":"f197ee8f60a6cde41abb7f3399971fea2a37fd2be9d312527194a634d04b651d"}
Jul 09 18:08:38.898540 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:38.898499    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerID="1541a4bd954325011ac47bc050f2b7fce9cadc9b730b563c2b79b13d8397d765" exitCode=0
Jul 09 18:08:38.898976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:38.898582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955" event={"ID":"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2","Type":"ContainerDied","Data":"1541a4bd954325011ac47bc050f2b7fce9cadc9b730b563c2b79b13d8397d765"}
Jul 09 18:08:46.938137 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:46.938103    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerID="dcf808d5b3ebe9a8f32fb1deaea956204ead4fa8302d059d665495131f0bd1cf" exitCode=0
Jul 09 18:08:46.938532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:46.938179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955" event={"ID":"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2","Type":"ContainerDied","Data":"dcf808d5b3ebe9a8f32fb1deaea956204ead4fa8302d059d665495131f0bd1cf"}
Jul 09 18:08:48.113212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.113181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:48.246180 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.246085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util\") pod \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") "
Jul 09 18:08:48.246180 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.246151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vktgw\" (UniqueName: \"kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw\") pod \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") "
Jul 09 18:08:48.246422 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.246192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle\") pod \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\" (UID: \"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2\") "
Jul 09 18:08:48.246980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.246939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle" (OuterVolumeSpecName: "bundle") pod "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" (UID: "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:48.248410 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.248383    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw" (OuterVolumeSpecName: "kube-api-access-vktgw") pod "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" (UID: "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2"). InnerVolumeSpecName "kube-api-access-vktgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:48.253536 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.253510    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util" (OuterVolumeSpecName: "util") pod "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" (UID: "ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:48.347144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.347108    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:48.347144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.347137    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vktgw\" (UniqueName: \"kubernetes.io/projected/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-kube-api-access-vktgw\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:48.347144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.347148    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:08:48.947214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.947183    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955"
Jul 09 18:08:48.947214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.947208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589x5955" event={"ID":"ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2","Type":"ContainerDied","Data":"5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff"}
Jul 09 18:08:48.947596 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:48.947269    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5817a9bd8a721fd9264779adcaaf6ecbe040c22abb61f041882f923d45947dff"
Jul 09 18:08:56.599225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599144    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"]
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599460    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="extract"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="extract"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599488    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="util"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599494    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="util"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599504    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="pull"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599510    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="pull"
Jul 09 18:08:56.599674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.599560    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae10136c-d8d9-4fa0-b3b7-e5dc6e27cca2" containerName="extract"
Jul 09 18:08:56.602395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.602378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.609884 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.609864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 18:08:56.609884 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.609874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 18:08:56.609884 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.609873    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 18:08:56.610085 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.609880    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dlnj5\""
Jul 09 18:08:56.619484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.619448    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"]
Jul 09 18:08:56.717061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.717026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0e98e9ed-1790-4397-ba3b-74662c5ca42e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.717256 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.717092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxlw\" (UniqueName: \"kubernetes.io/projected/0e98e9ed-1790-4397-ba3b-74662c5ca42e-kube-api-access-vrxlw\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.818398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.818362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0e98e9ed-1790-4397-ba3b-74662c5ca42e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.818616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.818443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxlw\" (UniqueName: \"kubernetes.io/projected/0e98e9ed-1790-4397-ba3b-74662c5ca42e-kube-api-access-vrxlw\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.820728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.820703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0e98e9ed-1790-4397-ba3b-74662c5ca42e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.834649 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.834622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxlw\" (UniqueName: \"kubernetes.io/projected/0e98e9ed-1790-4397-ba3b-74662c5ca42e-kube-api-access-vrxlw\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv\" (UID: \"0e98e9ed-1790-4397-ba3b-74662c5ca42e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:56.912613 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:56.912516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:08:57.067919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:57.067889    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"]
Jul 09 18:08:57.070510 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:08:57.070481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e98e9ed_1790_4397_ba3b_74662c5ca42e.slice/crio-b16e1566ff3934e6e23f77281c5ef28476229bc17bac61cec6e80a220d40ea7d WatchSource:0}: Error finding container b16e1566ff3934e6e23f77281c5ef28476229bc17bac61cec6e80a220d40ea7d: Status 404 returned error can't find the container with id b16e1566ff3934e6e23f77281c5ef28476229bc17bac61cec6e80a220d40ea7d
Jul 09 18:08:57.985025 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:57.984984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv" event={"ID":"0e98e9ed-1790-4397-ba3b-74662c5ca42e","Type":"ContainerStarted","Data":"b16e1566ff3934e6e23f77281c5ef28476229bc17bac61cec6e80a220d40ea7d"}
Jul 09 18:08:59.996299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:59.996252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv" event={"ID":"0e98e9ed-1790-4397-ba3b-74662c5ca42e","Type":"ContainerStarted","Data":"275d3925c10773fc20255f5f942c9163fb94742b3f4c773738cd9a86812d2189"}
Jul 09 18:08:59.996729 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:08:59.996326    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:09:00.047867 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.047801    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv" podStartSLOduration=1.553757636 podStartE2EDuration="4.047779692s" totalImagesPullingTime="2.494022056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:00.045063048 +0000 UTC m=+362.132312353" watchObservedRunningTime="2026-07-09 18:09:00.047779692 +0000 UTC m=+362.135028996"
Jul 09 18:09:00.863600 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.863563    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-6sppq"]
Jul 09 18:09:00.866919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.866895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:00.870512 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:00.870474    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"keda-operator-68bff59c-6sppq\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="028f3da6-13d4-49c0-a91c-d577db4b6e63" pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:00.870512 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:00.870495    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"keda-operator-dockercfg-z7l2q\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-z7l2q\"" type="*v1.Secret"
Jul 09 18:09:00.870863 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:00.870835    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"keda-ocp-cabundle\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" type="*v1.ConfigMap"
Jul 09 18:09:00.870975 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:00.870950    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"keda-operator-certs\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" type="*v1.Secret"
Jul 09 18:09:00.901273 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.901212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-6sppq"]
Jul 09 18:09:00.955563 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.955523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:00.955749 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.955578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-certificates\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:00.955749 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:00.955644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9vm\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-kube-api-access-zh9vm\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.057109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.057064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-certificates\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.057109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.057118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9vm\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-kube-api-access-zh9vm\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.057596 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.057166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.078446 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.078414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9vm\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-kube-api-access-zh9vm\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.613516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.613471    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"]
Jul 09 18:09:01.617081 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.617048    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.625024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.624992    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 09 18:09:01.639419 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.639329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"]
Jul 09 18:09:01.764056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.764010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-certificates\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.764056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.764066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.764385 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.764098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnf4s\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-kube-api-access-jnf4s\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.846401 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.846362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 09 18:09:01.849787 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.849754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/028f3da6-13d4-49c0-a91c-d577db4b6e63-certificates\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:01.865579 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.865487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-certificates\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.865579 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.865527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.865579 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.865555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnf4s\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-kube-api-access-jnf4s\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.866002 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.865982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.878904 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.878870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnf4s\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-kube-api-access-jnf4s\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:01.917105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.917067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-22qgs"]
Jul 09 18:09:01.920607 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.920586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:01.922956 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.922934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 09 18:09:01.933077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:01.933049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-22qgs"]
Jul 09 18:09:02.058406 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:02.058368    2565 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: failed to sync configmap cache: timed out waiting for the condition
Jul 09 18:09:02.058790 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:09:02.058454    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0 podName:028f3da6-13d4-49c0-a91c-d577db4b6e63 nodeName:}" failed. No retries permitted until 2026-07-09 18:09:02.558435928 +0000 UTC m=+364.645685210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0") pod "keda-operator-68bff59c-6sppq" (UID: "028f3da6-13d4-49c0-a91c-d577db4b6e63") : failed to sync configmap cache: timed out waiting for the condition
Jul 09 18:09:02.066801 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.066764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-certificates\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.066951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.066831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zcd\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-kube-api-access-99zcd\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.167914 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.167820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-certificates\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.167914 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.167869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zcd\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-kube-api-access-99zcd\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.188024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.187994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zcd\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-kube-api-access-99zcd\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.295017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.294986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-z7l2q\""
Jul 09 18:09:02.377631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.377595    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 18:09:02.381039 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.381001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5cbc4162-cd53-4d1b-9490-f9850b8e16cc-certificates\") pod \"keda-admission-5fbd5c4644-22qgs\" (UID: \"5cbc4162-cd53-4d1b-9490-f9850b8e16cc\") " pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.389150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.389116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d-certificates\") pod \"keda-metrics-apiserver-559f75f947-8nj5q\" (UID: \"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:02.529211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.529176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:02.531371 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.531337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:02.571715 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.571678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:02.572398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.572378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/028f3da6-13d4-49c0-a91c-d577db4b6e63-cabundle0\") pod \"keda-operator-68bff59c-6sppq\" (UID: \"028f3da6-13d4-49c0-a91c-d577db4b6e63\") " pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:02.677765 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.677733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:02.695800 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.695764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"]
Jul 09 18:09:02.699292 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:09:02.699259    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf2c4d5_21bf_434f_9ec6_733aa3c5ef4d.slice/crio-d1ca2777c6093da393edc15a89f882b920da1b746d7f30ef88042fef65651d78 WatchSource:0}: Error finding container d1ca2777c6093da393edc15a89f882b920da1b746d7f30ef88042fef65651d78: Status 404 returned error can't find the container with id d1ca2777c6093da393edc15a89f882b920da1b746d7f30ef88042fef65651d78
Jul 09 18:09:02.710457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.710422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-22qgs"]
Jul 09 18:09:02.714035 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:09:02.713995    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbc4162_cd53_4d1b_9490_f9850b8e16cc.slice/crio-c400d289f7e86a56107cde07591f2dbb67ee40252c1188f8a4067ae6c84e6ad6 WatchSource:0}: Error finding container c400d289f7e86a56107cde07591f2dbb67ee40252c1188f8a4067ae6c84e6ad6: Status 404 returned error can't find the container with id c400d289f7e86a56107cde07591f2dbb67ee40252c1188f8a4067ae6c84e6ad6
Jul 09 18:09:02.825909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:02.825831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-6sppq"]
Jul 09 18:09:02.828840 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:09:02.828815    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028f3da6_13d4_49c0_a91c_d577db4b6e63.slice/crio-910d3dcc86afcb0d078a9d7542308b1423c8672d95dea23c83ac8ad3d9bcce36 WatchSource:0}: Error finding container 910d3dcc86afcb0d078a9d7542308b1423c8672d95dea23c83ac8ad3d9bcce36: Status 404 returned error can't find the container with id 910d3dcc86afcb0d078a9d7542308b1423c8672d95dea23c83ac8ad3d9bcce36
Jul 09 18:09:03.010631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:03.010592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-6sppq" event={"ID":"028f3da6-13d4-49c0-a91c-d577db4b6e63","Type":"ContainerStarted","Data":"910d3dcc86afcb0d078a9d7542308b1423c8672d95dea23c83ac8ad3d9bcce36"}
Jul 09 18:09:03.011654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:03.011627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q" event={"ID":"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d","Type":"ContainerStarted","Data":"d1ca2777c6093da393edc15a89f882b920da1b746d7f30ef88042fef65651d78"}
Jul 09 18:09:03.012764 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:03.012725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs" event={"ID":"5cbc4162-cd53-4d1b-9490-f9850b8e16cc","Type":"ContainerStarted","Data":"c400d289f7e86a56107cde07591f2dbb67ee40252c1188f8a4067ae6c84e6ad6"}
Jul 09 18:09:05.024595 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:05.024554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs" event={"ID":"5cbc4162-cd53-4d1b-9490-f9850b8e16cc","Type":"ContainerStarted","Data":"d0e91985118b626b646daf1fbea1d90773a9edc70be2440a08407102b0c564bb"}
Jul 09 18:09:05.025031 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:05.024708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:05.042374 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:05.042312    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs" podStartSLOduration=2.530163697 podStartE2EDuration="4.042289866s" totalImagesPullingTime="1.512126169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:05.041763397 +0000 UTC m=+367.129012701" watchObservedRunningTime="2026-07-09 18:09:05.042289866 +0000 UTC m=+367.129539171"
Jul 09 18:09:06.031807 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:06.031769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q" event={"ID":"eaf2c4d5-21bf-434f-9ec6-733aa3c5ef4d","Type":"ContainerStarted","Data":"c46f6b9200bf8668002d349b55d8af16701f326b47c97c739680a68cd744ed73"}
Jul 09 18:09:06.032323 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:06.031913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:08.042735 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:08.042642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-6sppq" event={"ID":"028f3da6-13d4-49c0-a91c-d577db4b6e63","Type":"ContainerStarted","Data":"29cb49e2f0cacc9e08a365661bdd20b9a7c3b1ebcb558ebc2a41891dd32a176a"}
Jul 09 18:09:08.043164 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:08.042747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:09:08.064533 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:08.064482    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q" podStartSLOduration=4.371296381 podStartE2EDuration="7.064466347s" totalImagesPullingTime="2.693169966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:06.055759526 +0000 UTC m=+368.143008831" watchObservedRunningTime="2026-07-09 18:09:08.064466347 +0000 UTC m=+370.151715651"
Jul 09 18:09:08.064754 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:08.064623    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-6sppq" podStartSLOduration=3.160881173 podStartE2EDuration="8.064613826s" totalImagesPullingTime="4.903732653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:08.063953537 +0000 UTC m=+370.151202844" watchObservedRunningTime="2026-07-09 18:09:08.064613826 +0000 UTC m=+370.151863131"
Jul 09 18:09:17.041489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:17.041459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-8nj5q"
Jul 09 18:09:21.002625 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:21.002588    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nvkhv"
Jul 09 18:09:26.034600 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:26.034561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-22qgs"
Jul 09 18:09:29.049544 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:09:29.049502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-6sppq"
Jul 09 18:10:13.149576 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.149543    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:10:13.152788 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.152770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.156261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.156213    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 09 18:10:13.156261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.156256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-lgrhm\""
Jul 09 18:10:13.157010 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.156995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:10:13.161088 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.161071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:10:13.187127 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.187094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:10:13.249676 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.249643    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 18:10:13.252835 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.252819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.256750 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.256726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcldk\" (UniqueName: \"kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.256816 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.256795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.258338 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:13.258311    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"llmisvc-controller-manager-fbfcc4968-m62ct\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.258602 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:13.258581    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret"
Jul 09 18:10:13.258920 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:13.258894    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-controller-manager-dockercfg-ff5mv\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ff5mv\"" type="*v1.Secret"
Jul 09 18:10:13.264625 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.264605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 18:10:13.357851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.357820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.357863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.357909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thcb\" (UniqueName: \"kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.357929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcldk\" (UniqueName: \"kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.358019 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:13.357977    2565 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 09 18:10:13.358152 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:13.358088    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert podName:1e10fda8-a58a-4c7d-b8b4-c61de6b545bb nodeName:}" failed. No retries permitted until 2026-07-09 18:10:13.858070934 +0000 UTC m=+435.945320221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert") pod "kserve-controller-manager-748bdf6488-9nthj" (UID: "1e10fda8-a58a-4c7d-b8b4-c61de6b545bb") : secret "kserve-webhook-server-cert" not found
Jul 09 18:10:13.396165 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.396136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcldk\" (UniqueName: \"kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.458518 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.458483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thcb\" (UniqueName: \"kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.458687 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.458552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.473014 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.472978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thcb\" (UniqueName: \"kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:13.862382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.862297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:13.864795 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:13.864770    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") pod \"kserve-controller-manager-748bdf6488-9nthj\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") " pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:14.062948 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.062897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:14.200497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.200469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:10:14.202912 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:10:14.202883    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e10fda8_a58a_4c7d_b8b4_c61de6b545bb.slice/crio-a4f2fb7d9a815a2f531f99338dfc354204fafe7dfe754f87d786425ec4515675 WatchSource:0}: Error finding container a4f2fb7d9a815a2f531f99338dfc354204fafe7dfe754f87d786425ec4515675: Status 404 returned error can't find the container with id a4f2fb7d9a815a2f531f99338dfc354204fafe7dfe754f87d786425ec4515675
Jul 09 18:10:14.324442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.324409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" event={"ID":"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb","Type":"ContainerStarted","Data":"a4f2fb7d9a815a2f531f99338dfc354204fafe7dfe754f87d786425ec4515675"}
Jul 09 18:10:14.327937 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.327913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ff5mv\""
Jul 09 18:10:14.459635 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:14.459600    2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: failed to sync secret cache: timed out waiting for the condition
Jul 09 18:10:14.459785 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:10:14.459684    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert podName:73f547a2-9f2e-4b4e-822a-a22eb6f45de6 nodeName:}" failed. No retries permitted until 2026-07-09 18:10:14.9596645 +0000 UTC m=+437.046913786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert") pod "llmisvc-controller-manager-fbfcc4968-m62ct" (UID: "73f547a2-9f2e-4b4e-822a-a22eb6f45de6") : failed to sync secret cache: timed out waiting for the condition
Jul 09 18:10:14.565803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.565772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 18:10:14.972409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.972378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:14.975060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:14.975036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") pod \"llmisvc-controller-manager-fbfcc4968-m62ct\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") " pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:15.063043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:15.063007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:15.244471 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:15.244441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 18:10:15.247726 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:10:15.247673    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73f547a2_9f2e_4b4e_822a_a22eb6f45de6.slice/crio-aad683483efaeb98c961e3e851cf0a4511ecd310d4fd874b8b246aca1f5d9946 WatchSource:0}: Error finding container aad683483efaeb98c961e3e851cf0a4511ecd310d4fd874b8b246aca1f5d9946: Status 404 returned error can't find the container with id aad683483efaeb98c961e3e851cf0a4511ecd310d4fd874b8b246aca1f5d9946
Jul 09 18:10:15.330859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:15.330808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" event={"ID":"73f547a2-9f2e-4b4e-822a-a22eb6f45de6","Type":"ContainerStarted","Data":"aad683483efaeb98c961e3e851cf0a4511ecd310d4fd874b8b246aca1f5d9946"}
Jul 09 18:10:18.347160 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:18.347124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" event={"ID":"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb","Type":"ContainerStarted","Data":"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"}
Jul 09 18:10:18.347566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:18.347257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:18.366043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:18.365983    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" podStartSLOduration=2.3183010729999998 podStartE2EDuration="5.365964014s" totalImagesPullingTime="3.047662941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:18.364848672 +0000 UTC m=+440.452097970" watchObservedRunningTime="2026-07-09 18:10:18.365964014 +0000 UTC m=+440.453213318"
Jul 09 18:10:20.357976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:20.357884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" event={"ID":"73f547a2-9f2e-4b4e-822a-a22eb6f45de6","Type":"ContainerStarted","Data":"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"}
Jul 09 18:10:20.358387 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:20.358060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:20.383817 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:20.383770    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" podStartSLOduration=2.883347225 podStartE2EDuration="7.383750092s" totalImagesPullingTime="4.500402867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:20.382382806 +0000 UTC m=+442.469632110" watchObservedRunningTime="2026-07-09 18:10:20.383750092 +0000 UTC m=+442.470999397"
Jul 09 18:10:49.357881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:49.357850    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:10:51.364863 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:51.364830    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 18:10:52.740908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.740871    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"]
Jul 09 18:10:52.743860 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.743842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.746774 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.746745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f4j7t\""
Jul 09 18:10:52.746774 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.746771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:10:52.747970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.747955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:10:52.756006 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.755984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"]
Jul 09 18:10:52.891519 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.891463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.891519 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.891523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.891778 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.891562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.992984 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.992889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.992984 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.992935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.992984 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.992961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.993646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.993616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:52.993833 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:52.993670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:53.005028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.004999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:53.052797 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.052765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:53.184157 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.184132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"]
Jul 09 18:10:53.187282 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:10:53.187254    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfd0c0b_e18a_4c3f_b270_eca44cf3a973.slice/crio-09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5 WatchSource:0}: Error finding container 09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5: Status 404 returned error can't find the container with id 09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5
Jul 09 18:10:53.499315 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.499280    2565 generic.go:356] "Generic (PLEG): container finished" podID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerID="bb7cb940c719f8a0abb3c79f545c7f06e9902989a67657d1cf021b19c4829b7a" exitCode=0
Jul 09 18:10:53.499493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.499340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c" event={"ID":"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973","Type":"ContainerDied","Data":"bb7cb940c719f8a0abb3c79f545c7f06e9902989a67657d1cf021b19c4829b7a"}
Jul 09 18:10:53.499493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:53.499373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c" event={"ID":"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973","Type":"ContainerStarted","Data":"09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5"}
Jul 09 18:10:55.509659 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:55.509623    2565 generic.go:356] "Generic (PLEG): container finished" podID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerID="fd8746e9386abdaba0d9a762940e31f7530347a0276c07e544f3843e8e2c97f3" exitCode=0
Jul 09 18:10:55.510049 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:55.509674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c" event={"ID":"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973","Type":"ContainerDied","Data":"fd8746e9386abdaba0d9a762940e31f7530347a0276c07e544f3843e8e2c97f3"}
Jul 09 18:10:56.515766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:56.515731    2565 generic.go:356] "Generic (PLEG): container finished" podID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerID="73e65e1e377e2013112e77844c436259e40c6560827bdbd87c5cc3f01289559c" exitCode=0
Jul 09 18:10:56.516165 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:56.515824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c" event={"ID":"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973","Type":"ContainerDied","Data":"73e65e1e377e2013112e77844c436259e40c6560827bdbd87c5cc3f01289559c"}
Jul 09 18:10:57.651132 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.651107    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:57.835134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.835030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle\") pod \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") "
Jul 09 18:10:57.835134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.835100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh\") pod \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") "
Jul 09 18:10:57.835393 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.835146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util\") pod \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\" (UID: \"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973\") "
Jul 09 18:10:57.835798 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.835769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle" (OuterVolumeSpecName: "bundle") pod "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" (UID: "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:10:57.837155 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.837127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh" (OuterVolumeSpecName: "kube-api-access-skfzh") pod "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" (UID: "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973"). InnerVolumeSpecName "kube-api-access-skfzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:57.840670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.840631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util" (OuterVolumeSpecName: "util") pod "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" (UID: "7cfd0c0b-e18a-4c3f-b270-eca44cf3a973"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:10:57.936209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.936166    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:10:57.936209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.936200    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:10:57.936209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:57.936212    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/7cfd0c0b-e18a-4c3f-b270-eca44cf3a973-kube-api-access-skfzh\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:10:58.527444 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:58.527417    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c"
Jul 09 18:10:58.527444 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:58.527424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkn7c" event={"ID":"7cfd0c0b-e18a-4c3f-b270-eca44cf3a973","Type":"ContainerDied","Data":"09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5"}
Jul 09 18:10:58.527706 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:10:58.527496    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09606e4a701fb598a168510fdbc2c83d11232ac4144b4f5e2777a1b9fd81acb5"
Jul 09 18:11:09.517447 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517412    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7599f89d75-dfw7d"]
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517724    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="util"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517735    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="util"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517746    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="pull"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="pull"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517759    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="extract"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517764    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="extract"
Jul 09 18:11:09.517862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.517831    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfd0c0b-e18a-4c3f-b270-eca44cf3a973" containerName="extract"
Jul 09 18:11:09.521077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.521058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.541116 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.541080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7599f89d75-dfw7d"]
Jul 09 18:11:09.627397 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-trusted-ca-bundle\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627397 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgc6j\" (UniqueName: \"kubernetes.io/projected/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-kube-api-access-bgc6j\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-oauth-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-oauth-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.627621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.627596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-service-ca\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728199 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728199 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-oauth-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-service-ca\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-trusted-ca-bundle\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgc6j\" (UniqueName: \"kubernetes.io/projected/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-kube-api-access-bgc6j\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.728467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.728319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-oauth-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.729069 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.729040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-oauth-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.729069 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.729056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-service-ca\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.729265 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.729067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.729265 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.729152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-trusted-ca-bundle\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.730842 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.730800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-oauth-config\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.731029 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.731007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-console-serving-cert\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.740466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.740433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgc6j\" (UniqueName: \"kubernetes.io/projected/18ab3736-5d7a-4d25-87cb-b5f283bf4a9b-kube-api-access-bgc6j\") pod \"console-7599f89d75-dfw7d\" (UID: \"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b\") " pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.831200 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.831113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:09.998776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:09.998750    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7599f89d75-dfw7d"]
Jul 09 18:11:10.001359 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:11:10.001329    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ab3736_5d7a_4d25_87cb_b5f283bf4a9b.slice/crio-6acd738ffc04bf79612c6a40645a3962950c06ecdc2d1ba7a4fcb12ccc18ae84 WatchSource:0}: Error finding container 6acd738ffc04bf79612c6a40645a3962950c06ecdc2d1ba7a4fcb12ccc18ae84: Status 404 returned error can't find the container with id 6acd738ffc04bf79612c6a40645a3962950c06ecdc2d1ba7a4fcb12ccc18ae84
Jul 09 18:11:10.586178 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:10.586137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599f89d75-dfw7d" event={"ID":"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b","Type":"ContainerStarted","Data":"480c37fc85d80dad898f853d91ddbcca5b90309c9136b1f0fbbf0dd1fe015b52"}
Jul 09 18:11:10.586572 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:10.586192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599f89d75-dfw7d" event={"ID":"18ab3736-5d7a-4d25-87cb-b5f283bf4a9b","Type":"ContainerStarted","Data":"6acd738ffc04bf79612c6a40645a3962950c06ecdc2d1ba7a4fcb12ccc18ae84"}
Jul 09 18:11:10.634126 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:10.634060    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7599f89d75-dfw7d" podStartSLOduration=1.63404316 podStartE2EDuration="1.63404316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:10.631518184 +0000 UTC m=+492.718767490" watchObservedRunningTime="2026-07-09 18:11:10.63404316 +0000 UTC m=+492.721292464"
Jul 09 18:11:14.968504 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.968463    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"]
Jul 09 18:11:14.972203 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.972180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:14.975283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.975261    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:11:14.975757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.975743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f4j7t\""
Jul 09 18:11:14.976056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.976040    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:11:14.995687 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:14.995659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"]
Jul 09 18:11:15.072187 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.072152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.072372 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.072272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjb7\" (UniqueName: \"kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.072372 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.072311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.172770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.172732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.172946 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.172796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjb7\" (UniqueName: \"kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.172946 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.172825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.173149 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.173133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.173196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.173160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.185903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.185869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjb7\" (UniqueName: \"kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.282350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.282263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:15.422369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.420685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"]
Jul 09 18:11:15.610903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.610803    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf6487c3-559a-493e-afb7-e06620e66c80" containerID="6ca9a9390734463f6905680d2b70cf79da19ba5d36fc39b8c0bfde82bf4f29f2" exitCode=0
Jul 09 18:11:15.610903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.610872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl" event={"ID":"cf6487c3-559a-493e-afb7-e06620e66c80","Type":"ContainerDied","Data":"6ca9a9390734463f6905680d2b70cf79da19ba5d36fc39b8c0bfde82bf4f29f2"}
Jul 09 18:11:15.611086 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:15.610937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl" event={"ID":"cf6487c3-559a-493e-afb7-e06620e66c80","Type":"ContainerStarted","Data":"2a086e2f555c7a9c375a618bc782e5a386aa5a30002aaa7b4099f7297eae03b2"}
Jul 09 18:11:18.627757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:18.627723    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf6487c3-559a-493e-afb7-e06620e66c80" containerID="a2940d9ffd6fd8ff3afaf9479176d6c6230a82088ecc2a490045fe5eb5266c0f" exitCode=0
Jul 09 18:11:18.628225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:18.627811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl" event={"ID":"cf6487c3-559a-493e-afb7-e06620e66c80","Type":"ContainerDied","Data":"a2940d9ffd6fd8ff3afaf9479176d6c6230a82088ecc2a490045fe5eb5266c0f"}
Jul 09 18:11:19.635399 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:19.635354    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf6487c3-559a-493e-afb7-e06620e66c80" containerID="a664ade3a05f56a400693aca7012370492fb7bee84d7c106c76e0ad99d61a8f0" exitCode=0
Jul 09 18:11:19.635848 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:19.635443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl" event={"ID":"cf6487c3-559a-493e-afb7-e06620e66c80","Type":"ContainerDied","Data":"a664ade3a05f56a400693aca7012370492fb7bee84d7c106c76e0ad99d61a8f0"}
Jul 09 18:11:19.832612 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:19.832573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:19.832803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:19.832641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:19.838605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:19.838571    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:20.643940 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.643912    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7599f89d75-dfw7d"
Jul 09 18:11:20.704364 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.704329    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:11:20.789471 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.789443    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:20.826568 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.826532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util\") pod \"cf6487c3-559a-493e-afb7-e06620e66c80\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") "
Jul 09 18:11:20.826782 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.826596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle\") pod \"cf6487c3-559a-493e-afb7-e06620e66c80\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") "
Jul 09 18:11:20.826782 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.826678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkjb7\" (UniqueName: \"kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7\") pod \"cf6487c3-559a-493e-afb7-e06620e66c80\" (UID: \"cf6487c3-559a-493e-afb7-e06620e66c80\") "
Jul 09 18:11:20.827033 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.827005    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle" (OuterVolumeSpecName: "bundle") pod "cf6487c3-559a-493e-afb7-e06620e66c80" (UID: "cf6487c3-559a-493e-afb7-e06620e66c80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:11:20.828849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.828819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7" (OuterVolumeSpecName: "kube-api-access-jkjb7") pod "cf6487c3-559a-493e-afb7-e06620e66c80" (UID: "cf6487c3-559a-493e-afb7-e06620e66c80"). InnerVolumeSpecName "kube-api-access-jkjb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:20.831963 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.831873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util" (OuterVolumeSpecName: "util") pod "cf6487c3-559a-493e-afb7-e06620e66c80" (UID: "cf6487c3-559a-493e-afb7-e06620e66c80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:11:20.928219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.928121    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjb7\" (UniqueName: \"kubernetes.io/projected/cf6487c3-559a-493e-afb7-e06620e66c80-kube-api-access-jkjb7\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.928219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.928151    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.928219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:20.928161    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf6487c3-559a-493e-afb7-e06620e66c80-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:21.646581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:21.646552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl"
Jul 09 18:11:21.646581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:21.646570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftc2kl" event={"ID":"cf6487c3-559a-493e-afb7-e06620e66c80","Type":"ContainerDied","Data":"2a086e2f555c7a9c375a618bc782e5a386aa5a30002aaa7b4099f7297eae03b2"}
Jul 09 18:11:21.647112 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:21.646624    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a086e2f555c7a9c375a618bc782e5a386aa5a30002aaa7b4099f7297eae03b2"
Jul 09 18:11:43.183797 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.183756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"]
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184082    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="extract"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184092    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="extract"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184109    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="pull"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="pull"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184127    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="util"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184132    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="util"
Jul 09 18:11:43.184283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.184187    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6487c3-559a-493e-afb7-e06620e66c80" containerName="extract"
Jul 09 18:11:43.190400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.190379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.193778 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.193754    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gpcpk\""
Jul 09 18:11:43.194524 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.194509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 18:11:43.194590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.194513    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 18:11:43.195466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.195452    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:11:43.195529 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.195450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 18:11:43.215654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.215608    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"]
Jul 09 18:11:43.316981 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.316940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgs7q\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-kube-api-access-mgs7q\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317183 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317183 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c422460-a5ad-4b42-95a0-0fb105c1a60e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.317399 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.317352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418526 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418526 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c422460-a5ad-4b42-95a0-0fb105c1a60e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.418775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.418654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgs7q\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-kube-api-access-mgs7q\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.419283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.419222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.420874 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.420843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c422460-a5ad-4b42-95a0-0fb105c1a60e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.421076 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.421058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.421352 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.421333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.421420 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.421391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.428970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.428945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.430724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.430696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgs7q\" (UniqueName: \"kubernetes.io/projected/4c422460-a5ad-4b42-95a0-0fb105c1a60e-kube-api-access-mgs7q\") pod \"istiod-openshift-gateway-94bb8fbfd-4wpb9\" (UID: \"4c422460-a5ad-4b42-95a0-0fb105c1a60e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.499567 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.499535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:43.646651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.646558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"]
Jul 09 18:11:43.649372 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:11:43.649343    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c422460_a5ad_4b42_95a0_0fb105c1a60e.slice/crio-0eb96d9cf799f9e0d43b02568eafea8256e072d57c49bf766014f7971ca65baa WatchSource:0}: Error finding container 0eb96d9cf799f9e0d43b02568eafea8256e072d57c49bf766014f7971ca65baa: Status 404 returned error can't find the container with id 0eb96d9cf799f9e0d43b02568eafea8256e072d57c49bf766014f7971ca65baa
Jul 09 18:11:43.751870 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:43.751778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9" event={"ID":"4c422460-a5ad-4b42-95a0-0fb105c1a60e","Type":"ContainerStarted","Data":"0eb96d9cf799f9e0d43b02568eafea8256e072d57c49bf766014f7971ca65baa"}
Jul 09 18:11:45.731172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:45.731125    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75b7685c4d-h4d4p" podUID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" containerName="console" containerID="cri-o://8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627" gracePeriod=15
Jul 09 18:11:45.991342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:45.991257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:11:46.142350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142302    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142397    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmrb6\" (UniqueName: \"kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142502    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config\") pod \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\" (UID: \"a6669d28-e3d0-459f-844a-8e0b8226b9d1\") "
Jul 09 18:11:46.142905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142696    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:46.142905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142847    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:46.143012 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142964    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:46.143012 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.142998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config" (OuterVolumeSpecName: "console-config") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:46.144762 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.144734    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:46.144881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.144818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6" (OuterVolumeSpecName: "kube-api-access-fmrb6") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "kube-api-access-fmrb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:46.144881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.144837    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6669d28-e3d0-459f-844a-8e0b8226b9d1" (UID: "a6669d28-e3d0-459f-844a-8e0b8226b9d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:46.243968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243876    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrb6\" (UniqueName: \"kubernetes.io/projected/a6669d28-e3d0-459f-844a-8e0b8226b9d1-kube-api-access-fmrb6\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.243968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243922    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-trusted-ca-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.243968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243938    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.243968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243953    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.243968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243967    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.244304 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243980    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6669d28-e3d0-459f-844a-8e0b8226b9d1-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.244304 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.243994    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6669d28-e3d0-459f-844a-8e0b8226b9d1-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:11:46.771216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.771173    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" containerID="8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627" exitCode=0
Jul 09 18:11:46.771216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.771220    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b7685c4d-h4d4p"
Jul 09 18:11:46.771690 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.771224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b7685c4d-h4d4p" event={"ID":"a6669d28-e3d0-459f-844a-8e0b8226b9d1","Type":"ContainerDied","Data":"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"}
Jul 09 18:11:46.771690 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.771293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b7685c4d-h4d4p" event={"ID":"a6669d28-e3d0-459f-844a-8e0b8226b9d1","Type":"ContainerDied","Data":"d31c30fadfb5771febb22a3242149047a884e908814f6020e15d654382ee701a"}
Jul 09 18:11:46.771690 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.771317    2565 scope.go:117] "RemoveContainer" containerID="8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"
Jul 09 18:11:46.780226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.780196    2565 scope.go:117] "RemoveContainer" containerID="8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"
Jul 09 18:11:46.780589 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:11:46.780564    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627\": container with ID starting with 8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627 not found: ID does not exist" containerID="8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"
Jul 09 18:11:46.780683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.780605    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627"} err="failed to get container status \"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627\": rpc error: code = NotFound desc = could not find container \"8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627\": container with ID starting with 8531d8b166efd0ab69d787c41133366a6293d6713829b877659106fa330ac627 not found: ID does not exist"
Jul 09 18:11:46.791190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.791156    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:11:46.793606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:46.793577    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75b7685c4d-h4d4p"]
Jul 09 18:11:48.482222 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:48.482183    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" path="/var/lib/kubelet/pods/a6669d28-e3d0-459f-844a-8e0b8226b9d1/volumes"
Jul 09 18:11:57.693053 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:57.693011    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:11:57.693352 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:57.693081    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:11:57.832046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:57.832011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9" event={"ID":"4c422460-a5ad-4b42-95a0-0fb105c1a60e","Type":"ContainerStarted","Data":"6d4e9199d8a6aef600ff3546c0c6c4c452ce4a9460ab99749ed53dbbd6ef12de"}
Jul 09 18:11:57.832217 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:57.832098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:11:57.854921 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:57.854873    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9" podStartSLOduration=0.813396489 podStartE2EDuration="14.854856999s" totalImagesPullingTime="14.04146051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:57.853701912 +0000 UTC m=+539.940951215" watchObservedRunningTime="2026-07-09 18:11:57.854856999 +0000 UTC m=+539.942106302"
Jul 09 18:11:58.838340 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:58.838308    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-4wpb9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 18:11:58.838726 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:11:58.838360    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9" podUID="4c422460-a5ad-4b42-95a0-0fb105c1a60e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:12:01.837513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:01.837480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4wpb9"
Jul 09 18:12:02.393633 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.393594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"]
Jul 09 18:12:02.394043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.394024    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" containerName="console"
Jul 09 18:12:02.394144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.394045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" containerName="console"
Jul 09 18:12:02.394226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.394170    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6669d28-e3d0-459f-844a-8e0b8226b9d1" containerName="console"
Jul 09 18:12:02.398873 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.398849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.403772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.403750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-thznt\""
Jul 09 18:12:02.418938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.418914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"]
Jul 09 18:12:02.488616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff85a442-d24b-4a72-a758-81e518cfd1fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.488844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqlx\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-kube-api-access-cfqlx\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.489055 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.488843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589584 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589584 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff85a442-d24b-4a72-a758-81e518cfd1fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqlx\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-kube-api-access-cfqlx\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.589856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.589782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.590114 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.590042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.590454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.590422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff85a442-d24b-4a72-a758-81e518cfd1fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.590586 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.590489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.590586 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.590554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.590586 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.590573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.592207 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.592182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.592399 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.592381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.597996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.597969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqlx\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-kube-api-access-cfqlx\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.598079 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.597970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff85a442-d24b-4a72-a758-81e518cfd1fd-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-wqvjb\" (UID: \"ff85a442-d24b-4a72-a758-81e518cfd1fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.710260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.710215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:02.854298 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:02.854274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"]
Jul 09 18:12:02.856223 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:02.856197    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff85a442_d24b_4a72_a758_81e518cfd1fd.slice/crio-a607585dec615cd2ef000f5739d780e50627a19c4dc19e9e9fd5f4b6f41fac48 WatchSource:0}: Error finding container a607585dec615cd2ef000f5739d780e50627a19c4dc19e9e9fd5f4b6f41fac48: Status 404 returned error can't find the container with id a607585dec615cd2ef000f5739d780e50627a19c4dc19e9e9fd5f4b6f41fac48
Jul 09 18:12:03.860145 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:03.860110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" event={"ID":"ff85a442-d24b-4a72-a758-81e518cfd1fd","Type":"ContainerStarted","Data":"a607585dec615cd2ef000f5739d780e50627a19c4dc19e9e9fd5f4b6f41fac48"}
Jul 09 18:12:38.265331 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:38.265289    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:12:38.265629 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:38.265390    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:12:38.265629 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:38.265425    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:12:39.026772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:39.026736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" event={"ID":"ff85a442-d24b-4a72-a758-81e518cfd1fd","Type":"ContainerStarted","Data":"45d19395c85236eb3e2cdd5c703fb6bace879b39461eb68d70a9d0e7f90d4994"}
Jul 09 18:12:39.051207 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:39.051146    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podStartSLOduration=1.6449116620000002 podStartE2EDuration="37.051127371s" totalImagesPullingTime="35.406215709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:39.04867892 +0000 UTC m=+581.135928225" watchObservedRunningTime="2026-07-09 18:12:39.051127371 +0000 UTC m=+581.138376674"
Jul 09 18:12:39.711088 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:39.711049    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:39.716014 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:39.715987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:40.031580 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:40.031493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:40.032392 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:40.032374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:12:43.228409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.228371    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"]
Jul 09 18:12:43.232041 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.232022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.234814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.234795    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:12:43.234814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.234802    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:12:43.234996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.234870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f4j7t\""
Jul 09 18:12:43.243336 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.243310    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"]
Jul 09 18:12:43.329090 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.329046    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"]
Jul 09 18:12:43.332632 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.332605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.344137 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.344109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"]
Jul 09 18:12:43.349491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.349465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.349617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.349546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.349617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.349567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25vk\" (UniqueName: \"kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.450617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.450617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25vk\" (UniqueName: \"kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.450847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.450847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.450847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.450847 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.450799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.451195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.451177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.451195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.451188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.466696 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.466658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"]
Jul 09 18:12:43.470441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.470419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.481513 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.481448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25vk\" (UniqueName: \"kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.502950 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.502914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"]
Jul 09 18:12:43.541979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.541912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:43.551467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.551429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.551638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.551470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.551638 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.551616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.551850 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.551830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.552020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.552000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.564758 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.564732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.609862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.609309    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"]
Jul 09 18:12:43.615017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.614994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.627794 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.627763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"]
Jul 09 18:12:43.641815 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.641777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:43.652983 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.652946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.653141 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.653012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.653189 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.653144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mk8\" (UniqueName: \"kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.693887 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.693861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"]
Jul 09 18:12:43.697147 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:43.697110    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a93886_06af_40e1_b627_bc601903b313.slice/crio-a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417 WatchSource:0}: Error finding container a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417: Status 404 returned error can't find the container with id a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417
Jul 09 18:12:43.754584 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.754770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mk8\" (UniqueName: \"kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.754770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.754770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.754931 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.754931 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.754853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7fq\" (UniqueName: \"kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.755259 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.755217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.755304 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.755262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.765248 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.765207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mk8\" (UniqueName: \"kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.779509 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.779408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:43.800106 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.800051    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"]
Jul 09 18:12:43.801874 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:43.801842    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb99cf80_71e1_4f84_bfba_d485556fe55b.slice/crio-13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe WatchSource:0}: Error finding container 13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe: Status 404 returned error can't find the container with id 13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe
Jul 09 18:12:43.856094 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.856059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.856290 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.856111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7fq\" (UniqueName: \"kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.856290 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.856151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.856469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.856450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.856512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.856481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.865048 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.865018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7fq\" (UniqueName: \"kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.928122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.928093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:43.929641 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:43.929616    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"]
Jul 09 18:12:43.932009 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:43.931983    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962e3eff_a188_449c_a80a_bfc073dde87a.slice/crio-6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821 WatchSource:0}: Error finding container 6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821: Status 404 returned error can't find the container with id 6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821
Jul 09 18:12:44.052432 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.052397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s" event={"ID":"962e3eff-a188-449c-a80a-bfc073dde87a","Type":"ContainerStarted","Data":"6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821"}
Jul 09 18:12:44.054297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.054269    2565 generic.go:356] "Generic (PLEG): container finished" podID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerID="3a8d86f59c6ee298d39dd3b03419dde431425f9a50165e1b5335715d9cb78860" exitCode=0
Jul 09 18:12:44.054433 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.054349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z" event={"ID":"db99cf80-71e1-4f84-bfba-d485556fe55b","Type":"ContainerDied","Data":"3a8d86f59c6ee298d39dd3b03419dde431425f9a50165e1b5335715d9cb78860"}
Jul 09 18:12:44.054433 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.054397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z" event={"ID":"db99cf80-71e1-4f84-bfba-d485556fe55b","Type":"ContainerStarted","Data":"13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe"}
Jul 09 18:12:44.056258 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.056185    2565 generic.go:356] "Generic (PLEG): container finished" podID="89a93886-06af-40e1-b627-bc601903b313" containerID="e0cd89ae8370c1725b257ce329baa9ab6ec6a2ac01f276a4b04b0392c1c8a2d0" exitCode=0
Jul 09 18:12:44.056432 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.056258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c" event={"ID":"89a93886-06af-40e1-b627-bc601903b313","Type":"ContainerDied","Data":"e0cd89ae8370c1725b257ce329baa9ab6ec6a2ac01f276a4b04b0392c1c8a2d0"}
Jul 09 18:12:44.056432 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.056304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c" event={"ID":"89a93886-06af-40e1-b627-bc601903b313","Type":"ContainerStarted","Data":"a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417"}
Jul 09 18:12:44.086313 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:44.086287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"]
Jul 09 18:12:44.089289 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:44.089225    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aad395_73d1_4b27_a6b2_5031a952c851.slice/crio-919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a WatchSource:0}: Error finding container 919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a: Status 404 returned error can't find the container with id 919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a
Jul 09 18:12:45.063351 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.063310    2565 generic.go:356] "Generic (PLEG): container finished" podID="89a93886-06af-40e1-b627-bc601903b313" containerID="eb2788b364d21eb2921399a82316d9c585925e7c11f95c63dec2142465bc506c" exitCode=0
Jul 09 18:12:45.063821 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.063398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c" event={"ID":"89a93886-06af-40e1-b627-bc601903b313","Type":"ContainerDied","Data":"eb2788b364d21eb2921399a82316d9c585925e7c11f95c63dec2142465bc506c"}
Jul 09 18:12:45.065150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.065127    2565 generic.go:356] "Generic (PLEG): container finished" podID="962e3eff-a188-449c-a80a-bfc073dde87a" containerID="e0e24e961218c34e6084c785b02fb09ac05c06490a9f4ccae0a5869ef4c1ca4a" exitCode=0
Jul 09 18:12:45.065211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.065198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s" event={"ID":"962e3eff-a188-449c-a80a-bfc073dde87a","Type":"ContainerDied","Data":"e0e24e961218c34e6084c785b02fb09ac05c06490a9f4ccae0a5869ef4c1ca4a"}
Jul 09 18:12:45.067332 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.067306    2565 generic.go:356] "Generic (PLEG): container finished" podID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerID="af025f2bf14ce30a965a10e1c56a9e968c6fd83eb92f325d01196d3a9cd97707" exitCode=0
Jul 09 18:12:45.067440 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.067392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z" event={"ID":"db99cf80-71e1-4f84-bfba-d485556fe55b","Type":"ContainerDied","Data":"af025f2bf14ce30a965a10e1c56a9e968c6fd83eb92f325d01196d3a9cd97707"}
Jul 09 18:12:45.069134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.069111    2565 generic.go:356] "Generic (PLEG): container finished" podID="46aad395-73d1-4b27-a6b2-5031a952c851" containerID="2eb2140c75e89e3e508c1a9075fae81b8bd3a3cc2090703e049a57cdcda46e0d" exitCode=0
Jul 09 18:12:45.069188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.069152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5" event={"ID":"46aad395-73d1-4b27-a6b2-5031a952c851","Type":"ContainerDied","Data":"2eb2140c75e89e3e508c1a9075fae81b8bd3a3cc2090703e049a57cdcda46e0d"}
Jul 09 18:12:45.069225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:45.069187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5" event={"ID":"46aad395-73d1-4b27-a6b2-5031a952c851","Type":"ContainerStarted","Data":"919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a"}
Jul 09 18:12:46.076080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.076042    2565 generic.go:356] "Generic (PLEG): container finished" podID="89a93886-06af-40e1-b627-bc601903b313" containerID="58dc6555bdd639d0902bfb82395ad3a43f89b45fa50fff89b1e6d24a715c4ee1" exitCode=0
Jul 09 18:12:46.076567 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.076129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c" event={"ID":"89a93886-06af-40e1-b627-bc601903b313","Type":"ContainerDied","Data":"58dc6555bdd639d0902bfb82395ad3a43f89b45fa50fff89b1e6d24a715c4ee1"}
Jul 09 18:12:46.078003 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.077978    2565 generic.go:356] "Generic (PLEG): container finished" podID="962e3eff-a188-449c-a80a-bfc073dde87a" containerID="44d5c16e5e5e5671a1a1d9189fc34d9d1ef1149f4e4e12c3f6a3006c255a7eda" exitCode=0
Jul 09 18:12:46.078123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.078058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s" event={"ID":"962e3eff-a188-449c-a80a-bfc073dde87a","Type":"ContainerDied","Data":"44d5c16e5e5e5671a1a1d9189fc34d9d1ef1149f4e4e12c3f6a3006c255a7eda"}
Jul 09 18:12:46.080344 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.080316    2565 generic.go:356] "Generic (PLEG): container finished" podID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerID="67f6facba35c80ba770dc5fd59d8ba3406045cb37b196ea5647acc6035a17f5e" exitCode=0
Jul 09 18:12:46.080648 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.080396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z" event={"ID":"db99cf80-71e1-4f84-bfba-d485556fe55b","Type":"ContainerDied","Data":"67f6facba35c80ba770dc5fd59d8ba3406045cb37b196ea5647acc6035a17f5e"}
Jul 09 18:12:46.082105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.082084    2565 generic.go:356] "Generic (PLEG): container finished" podID="46aad395-73d1-4b27-a6b2-5031a952c851" containerID="5b6099db844adcbcdece1097a05c80d1762610141798ad54c9dd31939ff1a904" exitCode=0
Jul 09 18:12:46.082224 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:46.082155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5" event={"ID":"46aad395-73d1-4b27-a6b2-5031a952c851","Type":"ContainerDied","Data":"5b6099db844adcbcdece1097a05c80d1762610141798ad54c9dd31939ff1a904"}
Jul 09 18:12:47.088670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.088635    2565 generic.go:356] "Generic (PLEG): container finished" podID="962e3eff-a188-449c-a80a-bfc073dde87a" containerID="ac0fb76678b8e1d38e2b22651aaa2505d3ee6eb745265ff42d1b0c4919d3c326" exitCode=0
Jul 09 18:12:47.089061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.088713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s" event={"ID":"962e3eff-a188-449c-a80a-bfc073dde87a","Type":"ContainerDied","Data":"ac0fb76678b8e1d38e2b22651aaa2505d3ee6eb745265ff42d1b0c4919d3c326"}
Jul 09 18:12:47.090704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.090682    2565 generic.go:356] "Generic (PLEG): container finished" podID="46aad395-73d1-4b27-a6b2-5031a952c851" containerID="b2fdfe579371803232f9bc23f7ab60b9252baa559112a107e15374cfcbfe59f4" exitCode=0
Jul 09 18:12:47.090812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.090739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5" event={"ID":"46aad395-73d1-4b27-a6b2-5031a952c851","Type":"ContainerDied","Data":"b2fdfe579371803232f9bc23f7ab60b9252baa559112a107e15374cfcbfe59f4"}
Jul 09 18:12:47.240798 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.240768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:47.266833 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.266812    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:47.383979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.383893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util\") pod \"89a93886-06af-40e1-b627-bc601903b313\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") "
Jul 09 18:12:47.383979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.383961    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle\") pod \"db99cf80-71e1-4f84-bfba-d485556fe55b\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") "
Jul 09 18:12:47.384168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.383995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle\") pod \"89a93886-06af-40e1-b627-bc601903b313\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") "
Jul 09 18:12:47.384168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util\") pod \"db99cf80-71e1-4f84-bfba-d485556fe55b\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") "
Jul 09 18:12:47.384168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d25vk\" (UniqueName: \"kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk\") pod \"89a93886-06af-40e1-b627-bc601903b313\" (UID: \"89a93886-06af-40e1-b627-bc601903b313\") "
Jul 09 18:12:47.384415 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384153    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn\") pod \"db99cf80-71e1-4f84-bfba-d485556fe55b\" (UID: \"db99cf80-71e1-4f84-bfba-d485556fe55b\") "
Jul 09 18:12:47.384560 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle" (OuterVolumeSpecName: "bundle") pod "89a93886-06af-40e1-b627-bc601903b313" (UID: "89a93886-06af-40e1-b627-bc601903b313"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.384634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle" (OuterVolumeSpecName: "bundle") pod "db99cf80-71e1-4f84-bfba-d485556fe55b" (UID: "db99cf80-71e1-4f84-bfba-d485556fe55b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.384827 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384805    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.384827 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.384827    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.386777 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.386754    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk" (OuterVolumeSpecName: "kube-api-access-d25vk") pod "89a93886-06af-40e1-b627-bc601903b313" (UID: "89a93886-06af-40e1-b627-bc601903b313"). InnerVolumeSpecName "kube-api-access-d25vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:47.386887 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.386792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn" (OuterVolumeSpecName: "kube-api-access-z7bmn") pod "db99cf80-71e1-4f84-bfba-d485556fe55b" (UID: "db99cf80-71e1-4f84-bfba-d485556fe55b"). InnerVolumeSpecName "kube-api-access-z7bmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:47.389358 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.389327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util" (OuterVolumeSpecName: "util") pod "89a93886-06af-40e1-b627-bc601903b313" (UID: "89a93886-06af-40e1-b627-bc601903b313"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.389818 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.389796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util" (OuterVolumeSpecName: "util") pod "db99cf80-71e1-4f84-bfba-d485556fe55b" (UID: "db99cf80-71e1-4f84-bfba-d485556fe55b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.485303 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.485273    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db99cf80-71e1-4f84-bfba-d485556fe55b-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.485303 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.485300    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d25vk\" (UniqueName: \"kubernetes.io/projected/89a93886-06af-40e1-b627-bc601903b313-kube-api-access-d25vk\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.485303 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.485310    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/db99cf80-71e1-4f84-bfba-d485556fe55b-kube-api-access-z7bmn\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.485514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:47.485319    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89a93886-06af-40e1-b627-bc601903b313-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.097740 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.097711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c"
Jul 09 18:12:48.098204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.097731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7zx2c" event={"ID":"89a93886-06af-40e1-b627-bc601903b313","Type":"ContainerDied","Data":"a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417"}
Jul 09 18:12:48.098204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.097810    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f824dfdcbd1200ce006e226571c7c56f04c5b774cbe58416adf6ee75a38417"
Jul 09 18:12:48.099684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.099662    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z"
Jul 09 18:12:48.099812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.099701    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabb944z" event={"ID":"db99cf80-71e1-4f84-bfba-d485556fe55b","Type":"ContainerDied","Data":"13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe"}
Jul 09 18:12:48.099812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.099743    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13bc9ac4b0f93effe1c49af5a53f4899067cf31e134c65f5c5fc52ab7706b7fe"
Jul 09 18:12:48.252598 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.252575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:48.273286 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.273260    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:48.291315 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle\") pod \"962e3eff-a188-449c-a80a-bfc073dde87a\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") "
Jul 09 18:12:48.291489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util\") pod \"962e3eff-a188-449c-a80a-bfc073dde87a\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") "
Jul 09 18:12:48.291489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7fq\" (UniqueName: \"kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq\") pod \"46aad395-73d1-4b27-a6b2-5031a952c851\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") "
Jul 09 18:12:48.291489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle\") pod \"46aad395-73d1-4b27-a6b2-5031a952c851\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") "
Jul 09 18:12:48.291654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291494    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2mk8\" (UniqueName: \"kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8\") pod \"962e3eff-a188-449c-a80a-bfc073dde87a\" (UID: \"962e3eff-a188-449c-a80a-bfc073dde87a\") "
Jul 09 18:12:48.291654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.291537    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util\") pod \"46aad395-73d1-4b27-a6b2-5031a952c851\" (UID: \"46aad395-73d1-4b27-a6b2-5031a952c851\") "
Jul 09 18:12:48.292058 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.292019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle" (OuterVolumeSpecName: "bundle") pod "962e3eff-a188-449c-a80a-bfc073dde87a" (UID: "962e3eff-a188-449c-a80a-bfc073dde87a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:48.292305 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.292260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle" (OuterVolumeSpecName: "bundle") pod "46aad395-73d1-4b27-a6b2-5031a952c851" (UID: "46aad395-73d1-4b27-a6b2-5031a952c851"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:48.293823 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.293802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq" (OuterVolumeSpecName: "kube-api-access-qp7fq") pod "46aad395-73d1-4b27-a6b2-5031a952c851" (UID: "46aad395-73d1-4b27-a6b2-5031a952c851"). InnerVolumeSpecName "kube-api-access-qp7fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:48.294172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.294148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8" (OuterVolumeSpecName: "kube-api-access-x2mk8") pod "962e3eff-a188-449c-a80a-bfc073dde87a" (UID: "962e3eff-a188-449c-a80a-bfc073dde87a"). InnerVolumeSpecName "kube-api-access-x2mk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:48.297404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.297376    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util" (OuterVolumeSpecName: "util") pod "46aad395-73d1-4b27-a6b2-5031a952c851" (UID: "46aad395-73d1-4b27-a6b2-5031a952c851"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:48.297632 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.297611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util" (OuterVolumeSpecName: "util") pod "962e3eff-a188-449c-a80a-bfc073dde87a" (UID: "962e3eff-a188-449c-a80a-bfc073dde87a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392835    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392871    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mk8\" (UniqueName: \"kubernetes.io/projected/962e3eff-a188-449c-a80a-bfc073dde87a-kube-api-access-x2mk8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392882    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46aad395-73d1-4b27-a6b2-5031a952c851-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392893    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392900    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/962e3eff-a188-449c-a80a-bfc073dde87a-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:48.392926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:48.392908    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7fq\" (UniqueName: \"kubernetes.io/projected/46aad395-73d1-4b27-a6b2-5031a952c851-kube-api-access-qp7fq\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:12:49.107497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.107467    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s"
Jul 09 18:12:49.107497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.107478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36zrn2s" event={"ID":"962e3eff-a188-449c-a80a-bfc073dde87a","Type":"ContainerDied","Data":"6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821"}
Jul 09 18:12:49.107970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.107519    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3529ed346952856d44903754a5be424648308964f5a7272356f96bb208b821"
Jul 09 18:12:49.109400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.109375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5" event={"ID":"46aad395-73d1-4b27-a6b2-5031a952c851","Type":"ContainerDied","Data":"919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a"}
Jul 09 18:12:49.109400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.109393    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76l4lp5"
Jul 09 18:12:49.109585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:49.109418    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919afd8c06aac62b34984a8c21d3b786e8f46794784e94216b80ca25d5e2583a"
Jul 09 18:12:55.288162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288125    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"]
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288473    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="pull"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="pull"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288497    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288503    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288509    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288514    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288519    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="extract"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288525    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="extract"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288535    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288540    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="util"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288550    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="pull"
Jul 09 18:12:55.288551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288555    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="pull"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288562    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288567    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288578    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288585    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288591    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288599    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="pull"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288604    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="pull"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288613    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="util"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288618    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="util"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288627    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="pull"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288634    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="pull"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288686    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="962e3eff-a188-449c-a80a-bfc073dde87a" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288693    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46aad395-73d1-4b27-a6b2-5031a952c851" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288701    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a93886-06af-40e1-b627-bc601903b313" containerName="extract"
Jul 09 18:12:55.288890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.288707    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db99cf80-71e1-4f84-bfba-d485556fe55b" containerName="extract"
Jul 09 18:12:55.291722 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.291705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:12:55.294583 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.294561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-45vd7\""
Jul 09 18:12:55.294727 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.294711    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 18:12:55.295450 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.295437    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 18:12:55.308821 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.308796    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"]
Jul 09 18:12:55.353601 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.353559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj556\" (UniqueName: \"kubernetes.io/projected/caef0821-ceef-4284-b1a9-b5ea5ec283d0-kube-api-access-pj556\") pod \"limitador-operator-controller-manager-6f9f468797-8kx2v\" (UID: \"caef0821-ceef-4284-b1a9-b5ea5ec283d0\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:12:55.454075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.454039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj556\" (UniqueName: \"kubernetes.io/projected/caef0821-ceef-4284-b1a9-b5ea5ec283d0-kube-api-access-pj556\") pod \"limitador-operator-controller-manager-6f9f468797-8kx2v\" (UID: \"caef0821-ceef-4284-b1a9-b5ea5ec283d0\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:12:55.465699 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.465662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj556\" (UniqueName: \"kubernetes.io/projected/caef0821-ceef-4284-b1a9-b5ea5ec283d0-kube-api-access-pj556\") pod \"limitador-operator-controller-manager-6f9f468797-8kx2v\" (UID: \"caef0821-ceef-4284-b1a9-b5ea5ec283d0\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:12:55.601816 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.601725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:12:55.743692 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:55.743662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"]
Jul 09 18:12:55.745409 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:12:55.745384    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaef0821_ceef_4284_b1a9_b5ea5ec283d0.slice/crio-1c8846188c56a69ed5a4b36bfda72839faf75998c9f7ebe9f6cf19e17ee3f1ee WatchSource:0}: Error finding container 1c8846188c56a69ed5a4b36bfda72839faf75998c9f7ebe9f6cf19e17ee3f1ee: Status 404 returned error can't find the container with id 1c8846188c56a69ed5a4b36bfda72839faf75998c9f7ebe9f6cf19e17ee3f1ee
Jul 09 18:12:56.145971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:56.145930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v" event={"ID":"caef0821-ceef-4284-b1a9-b5ea5ec283d0","Type":"ContainerStarted","Data":"1c8846188c56a69ed5a4b36bfda72839faf75998c9f7ebe9f6cf19e17ee3f1ee"}
Jul 09 18:12:58.509564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.509540    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:12:58.509977 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.509888    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:12:58.513787 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.513766    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:12:58.514250 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.514220    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:12:58.520431 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.520411    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:12:58.521045 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:58.521027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:12:59.164768 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:59.164728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v" event={"ID":"caef0821-ceef-4284-b1a9-b5ea5ec283d0","Type":"ContainerStarted","Data":"615f045266f7ebd64a36edb92b7af47048f212a655d1ba1077f35d6b5706d8d7"}
Jul 09 18:12:59.164961 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:12:59.164878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:13:10.172610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:10.172574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v"
Jul 09 18:13:10.203100 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:10.203044    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-8kx2v" podStartSLOduration=12.399360459 podStartE2EDuration="15.203025744s" totalImagesPullingTime="2.803665285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:59.190067882 +0000 UTC m=+601.277317187" watchObservedRunningTime="2026-07-09 18:13:10.203025744 +0000 UTC m=+612.290275047"
Jul 09 18:13:53.075409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.075371    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:13:53.078784 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.078754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.082063 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.082039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gv6vk\""
Jul 09 18:13:53.082397 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.082379    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 18:13:53.092722 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.092697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:13:53.139284 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.139250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp52t\" (UniqueName: \"kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.139457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.139296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.143573 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.143542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:13:53.240060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.240028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp52t\" (UniqueName: \"kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.240060 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.240063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.240703 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.240683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.250351 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.250324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp52t\" (UniqueName: \"kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t\") pod \"limitador-limitador-ff68ff5cb-nhbx7\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.391214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.391103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:53.553389 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.553365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:13:53.554153 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:13:53.554129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66574e6f_9b16_448f_8f2a_419bfa69c096.slice/crio-3ce045d53a0eccd9b5762879c7c9ab5ed7b2fe7a1f518da03ff8660e4bf7f78a WatchSource:0}: Error finding container 3ce045d53a0eccd9b5762879c7c9ab5ed7b2fe7a1f518da03ff8660e4bf7f78a: Status 404 returned error can't find the container with id 3ce045d53a0eccd9b5762879c7c9ab5ed7b2fe7a1f518da03ff8660e4bf7f78a
Jul 09 18:13:53.556520 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:53.556505    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:13:54.438428 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:54.438387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" event={"ID":"66574e6f-9b16-448f-8f2a-419bfa69c096","Type":"ContainerStarted","Data":"3ce045d53a0eccd9b5762879c7c9ab5ed7b2fe7a1f518da03ff8660e4bf7f78a"}
Jul 09 18:13:58.462556 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:58.462517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" event={"ID":"66574e6f-9b16-448f-8f2a-419bfa69c096","Type":"ContainerStarted","Data":"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"}
Jul 09 18:13:58.463039 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:58.462604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:13:58.481859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:13:58.481807    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" podStartSLOduration=1.077670288 podStartE2EDuration="5.481788303s" totalImagesPullingTime="4.404118015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:58.480891059 +0000 UTC m=+660.568140364" watchObservedRunningTime="2026-07-09 18:13:58.481788303 +0000 UTC m=+660.569037608"
Jul 09 18:14:09.468196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:09.468161    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:14:09.639025 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:09.638994    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:14:09.639225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:09.639186    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" podUID="66574e6f-9b16-448f-8f2a-419bfa69c096" containerName="limitador" containerID="cri-o://8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc" gracePeriod=30
Jul 09 18:14:10.185269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.185227    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:14:10.288454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.288373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file\") pod \"66574e6f-9b16-448f-8f2a-419bfa69c096\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") "
Jul 09 18:14:10.288454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.288421    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp52t\" (UniqueName: \"kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t\") pod \"66574e6f-9b16-448f-8f2a-419bfa69c096\" (UID: \"66574e6f-9b16-448f-8f2a-419bfa69c096\") "
Jul 09 18:14:10.288739 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.288717    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file" (OuterVolumeSpecName: "config-file") pod "66574e6f-9b16-448f-8f2a-419bfa69c096" (UID: "66574e6f-9b16-448f-8f2a-419bfa69c096"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:14:10.290578 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.290557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t" (OuterVolumeSpecName: "kube-api-access-lp52t") pod "66574e6f-9b16-448f-8f2a-419bfa69c096" (UID: "66574e6f-9b16-448f-8f2a-419bfa69c096"). InnerVolumeSpecName "kube-api-access-lp52t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:14:10.389970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.389936    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66574e6f-9b16-448f-8f2a-419bfa69c096-config-file\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:14:10.389970 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.389965    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp52t\" (UniqueName: \"kubernetes.io/projected/66574e6f-9b16-448f-8f2a-419bfa69c096-kube-api-access-lp52t\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:14:10.523530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.523499    2565 generic.go:356] "Generic (PLEG): container finished" podID="66574e6f-9b16-448f-8f2a-419bfa69c096" containerID="8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc" exitCode=0
Jul 09 18:14:10.524020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.523554    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"
Jul 09 18:14:10.524020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.523582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" event={"ID":"66574e6f-9b16-448f-8f2a-419bfa69c096","Type":"ContainerDied","Data":"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"}
Jul 09 18:14:10.524020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.523627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7" event={"ID":"66574e6f-9b16-448f-8f2a-419bfa69c096","Type":"ContainerDied","Data":"3ce045d53a0eccd9b5762879c7c9ab5ed7b2fe7a1f518da03ff8660e4bf7f78a"}
Jul 09 18:14:10.524020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.523649    2565 scope.go:117] "RemoveContainer" containerID="8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"
Jul 09 18:14:10.532374 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.532355    2565 scope.go:117] "RemoveContainer" containerID="8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"
Jul 09 18:14:10.532635 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:14:10.532614    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc\": container with ID starting with 8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc not found: ID does not exist" containerID="8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"
Jul 09 18:14:10.532682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.532645    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc"} err="failed to get container status \"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc\": rpc error: code = NotFound desc = could not find container \"8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc\": container with ID starting with 8dbe0a3e2cf41cb6b0b65b7f9fcff39360360f0dc752b6fb8da05f7e2ecc2fdc not found: ID does not exist"
Jul 09 18:14:10.543144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.543033    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:14:10.544947 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:10.544924    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-nhbx7"]
Jul 09 18:14:12.482321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:12.482289    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66574e6f-9b16-448f-8f2a-419bfa69c096" path="/var/lib/kubelet/pods/66574e6f-9b16-448f-8f2a-419bfa69c096/volumes"
Jul 09 18:14:26.807289 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.807252    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:14:26.807825 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.807473    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" podUID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" containerName="manager" containerID="cri-o://a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb" gracePeriod=10
Jul 09 18:14:26.832028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.832000    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jqrkk"]
Jul 09 18:14:26.832379 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.832366    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66574e6f-9b16-448f-8f2a-419bfa69c096" containerName="limitador"
Jul 09 18:14:26.832438 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.832381    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66574e6f-9b16-448f-8f2a-419bfa69c096" containerName="limitador"
Jul 09 18:14:26.832438 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.832430    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66574e6f-9b16-448f-8f2a-419bfa69c096" containerName="limitador"
Jul 09 18:14:26.835403 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.835387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:26.845227 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.845204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jqrkk"]
Jul 09 18:14:26.933184 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.933158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcwg\" (UniqueName: \"kubernetes.io/projected/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-kube-api-access-xpcwg\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:26.933344 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:26.933199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-cert\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.033951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.033903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcwg\" (UniqueName: \"kubernetes.io/projected/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-kube-api-access-xpcwg\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.034141 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.033962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-cert\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.036408 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.036381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-cert\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.044003 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.043951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcwg\" (UniqueName: \"kubernetes.io/projected/6fe0bac6-5817-4f9b-ac0a-af163f0c81d7-kube-api-access-xpcwg\") pod \"kserve-controller-manager-748bdf6488-jqrkk\" (UID: \"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7\") " pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.065204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.065146    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:14:27.134986 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.134954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcldk\" (UniqueName: \"kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk\") pod \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") "
Jul 09 18:14:27.135150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.135023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") pod \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\" (UID: \"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb\") "
Jul 09 18:14:27.137207 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.137177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk" (OuterVolumeSpecName: "kube-api-access-gcldk") pod "1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" (UID: "1e10fda8-a58a-4c7d-b8b4-c61de6b545bb"). InnerVolumeSpecName "kube-api-access-gcldk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:14:27.137207 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.137177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert" (OuterVolumeSpecName: "cert") pod "1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" (UID: "1e10fda8-a58a-4c7d-b8b4-c61de6b545bb"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:14:27.194576 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.194525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:27.235610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.235579    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:14:27.235610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.235606    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcldk\" (UniqueName: \"kubernetes.io/projected/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb-kube-api-access-gcldk\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:14:27.325243 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.325143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jqrkk"]
Jul 09 18:14:27.328147 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:14:27.328116    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe0bac6_5817_4f9b_ac0a_af163f0c81d7.slice/crio-90673a584a8baa84e405089fdbdd660139feab8bf744e3004b022d66ca21e90e WatchSource:0}: Error finding container 90673a584a8baa84e405089fdbdd660139feab8bf744e3004b022d66ca21e90e: Status 404 returned error can't find the container with id 90673a584a8baa84e405089fdbdd660139feab8bf744e3004b022d66ca21e90e
Jul 09 18:14:27.610523 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.610434    2565 generic.go:356] "Generic (PLEG): container finished" podID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" containerID="a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb" exitCode=0
Jul 09 18:14:27.610523 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.610479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-9nthj"
Jul 09 18:14:27.610766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.610522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" event={"ID":"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb","Type":"ContainerDied","Data":"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"}
Jul 09 18:14:27.610766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.610577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-9nthj" event={"ID":"1e10fda8-a58a-4c7d-b8b4-c61de6b545bb","Type":"ContainerDied","Data":"a4f2fb7d9a815a2f531f99338dfc354204fafe7dfe754f87d786425ec4515675"}
Jul 09 18:14:27.610766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.610606    2565 scope.go:117] "RemoveContainer" containerID="a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"
Jul 09 18:14:27.611795 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.611771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk" event={"ID":"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7","Type":"ContainerStarted","Data":"90673a584a8baa84e405089fdbdd660139feab8bf744e3004b022d66ca21e90e"}
Jul 09 18:14:27.619186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.619167    2565 scope.go:117] "RemoveContainer" containerID="a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"
Jul 09 18:14:27.619442 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:14:27.619425    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb\": container with ID starting with a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb not found: ID does not exist" containerID="a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"
Jul 09 18:14:27.619495 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.619451    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb"} err="failed to get container status \"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb\": rpc error: code = NotFound desc = could not find container \"a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb\": container with ID starting with a305a35fd8bcf6429e881eed36782d7d623c61a82009424548b751d90e9024fb not found: ID does not exist"
Jul 09 18:14:27.632196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.632176    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:14:27.636039 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:27.636019    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-9nthj"]
Jul 09 18:14:28.482437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:28.482404    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" path="/var/lib/kubelet/pods/1e10fda8-a58a-4c7d-b8b4-c61de6b545bb/volumes"
Jul 09 18:14:28.618140 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:28.618101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk" event={"ID":"6fe0bac6-5817-4f9b-ac0a-af163f0c81d7","Type":"ContainerStarted","Data":"f4e3a15f4a7667407fafa33fdaaab30a077d3fc7134cabc25b4af55a9e200267"}
Jul 09 18:14:28.618368 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:28.618168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:14:28.639534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:28.639481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk" podStartSLOduration=2.221394396 podStartE2EDuration="2.639464773s" totalImagesPullingTime="418.070377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:28.637830639 +0000 UTC m=+690.725079945" watchObservedRunningTime="2026-07-09 18:14:28.639464773 +0000 UTC m=+690.726714077"
Jul 09 18:14:59.633801 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:14:59.633769    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-748bdf6488-jqrkk"
Jul 09 18:15:00.714591 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.714558    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-jpp7h"]
Jul 09 18:15:00.714995 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.714917    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" containerName="manager"
Jul 09 18:15:00.714995 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.714929    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" containerName="manager"
Jul 09 18:15:00.714995 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.714989    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e10fda8-a58a-4c7d-b8b4-c61de6b545bb" containerName="manager"
Jul 09 18:15:00.719380 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.719362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.722101 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.722079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 18:15:00.722260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.722162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tjvpz\""
Jul 09 18:15:00.731363 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.731337    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jpp7h"]
Jul 09 18:15:00.739742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.739708    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-chmc9"]
Jul 09 18:15:00.743105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.743083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.745634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.745616    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:15:00.745728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.745656    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rr2cc\""
Jul 09 18:15:00.755631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.755610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-chmc9"]
Jul 09 18:15:00.801905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.801868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzb72\" (UniqueName: \"kubernetes.io/projected/c3283ae9-c571-46cd-b755-6649389b912d-kube-api-access-mzb72\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.802139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.802088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422f8f1-a005-443e-bda7-4d0e12b5158c-tls-certs\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.802139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.802128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3283ae9-c571-46cd-b755-6649389b912d-cert\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.802307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.802181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2w9\" (UniqueName: \"kubernetes.io/projected/c422f8f1-a005-443e-bda7-4d0e12b5158c-kube-api-access-hq2w9\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.902612 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.902575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2w9\" (UniqueName: \"kubernetes.io/projected/c422f8f1-a005-443e-bda7-4d0e12b5158c-kube-api-access-hq2w9\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.902819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.902627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzb72\" (UniqueName: \"kubernetes.io/projected/c3283ae9-c571-46cd-b755-6649389b912d-kube-api-access-mzb72\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.902819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.902703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422f8f1-a005-443e-bda7-4d0e12b5158c-tls-certs\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.902819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.902730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3283ae9-c571-46cd-b755-6649389b912d-cert\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.905223 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.905195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422f8f1-a005-443e-bda7-4d0e12b5158c-tls-certs\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.905362 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.905283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3283ae9-c571-46cd-b755-6649389b912d-cert\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:00.910661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.910638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2w9\" (UniqueName: \"kubernetes.io/projected/c422f8f1-a005-443e-bda7-4d0e12b5158c-kube-api-access-hq2w9\") pod \"model-serving-api-fd65d7d6-jpp7h\" (UID: \"c422f8f1-a005-443e-bda7-4d0e12b5158c\") " pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:00.910905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:00.910883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzb72\" (UniqueName: \"kubernetes.io/projected/c3283ae9-c571-46cd-b755-6649389b912d-kube-api-access-mzb72\") pod \"odh-model-controller-84b54b7f97-chmc9\" (UID: \"c3283ae9-c571-46cd-b755-6649389b912d\") " pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:01.030780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.030677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:01.053874 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.053845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:01.182498 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.182428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jpp7h"]
Jul 09 18:15:01.184577 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:15:01.184548    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc422f8f1_a005_443e_bda7_4d0e12b5158c.slice/crio-124a6d9690195cd9c6657c89a49040930b232c07ddd8150acd41705ce92b6225 WatchSource:0}: Error finding container 124a6d9690195cd9c6657c89a49040930b232c07ddd8150acd41705ce92b6225: Status 404 returned error can't find the container with id 124a6d9690195cd9c6657c89a49040930b232c07ddd8150acd41705ce92b6225
Jul 09 18:15:01.200676 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.200652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-chmc9"]
Jul 09 18:15:01.203345 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:15:01.203313    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3283ae9_c571_46cd_b755_6649389b912d.slice/crio-e270a6a1454e8089fa061be9fa39451019561a411c158470cfd4a61288fccdad WatchSource:0}: Error finding container e270a6a1454e8089fa061be9fa39451019561a411c158470cfd4a61288fccdad: Status 404 returned error can't find the container with id e270a6a1454e8089fa061be9fa39451019561a411c158470cfd4a61288fccdad
Jul 09 18:15:01.785444 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.785412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jpp7h" event={"ID":"c422f8f1-a005-443e-bda7-4d0e12b5158c","Type":"ContainerStarted","Data":"124a6d9690195cd9c6657c89a49040930b232c07ddd8150acd41705ce92b6225"}
Jul 09 18:15:01.786529 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:01.786503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-chmc9" event={"ID":"c3283ae9-c571-46cd-b755-6649389b912d","Type":"ContainerStarted","Data":"e270a6a1454e8089fa061be9fa39451019561a411c158470cfd4a61288fccdad"}
Jul 09 18:15:02.795723 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:02.795689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jpp7h" event={"ID":"c422f8f1-a005-443e-bda7-4d0e12b5158c","Type":"ContainerStarted","Data":"6ecd7c276d774141a6fe7eb7a5dc70e0c87895bdb4339bb7fe765524917fa28e"}
Jul 09 18:15:02.796155 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:02.795778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:02.824099 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:02.824045    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-jpp7h" podStartSLOduration=1.295636737 podStartE2EDuration="2.824025312s" totalImagesPullingTime="1.528388575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:02.820522273 +0000 UTC m=+724.907771578" watchObservedRunningTime="2026-07-09 18:15:02.824025312 +0000 UTC m=+724.911274620"
Jul 09 18:15:04.808075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:04.808038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-chmc9" event={"ID":"c3283ae9-c571-46cd-b755-6649389b912d","Type":"ContainerStarted","Data":"c428b5c63d823db6439ac34116053fb36ea92c42af90e536c9e05a49e82c97db"}
Jul 09 18:15:04.808494 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:04.808101    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:15:04.828908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:04.828861    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-chmc9" podStartSLOduration=1.878744571 podStartE2EDuration="4.828844696s" totalImagesPullingTime="2.950100125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:04.827616306 +0000 UTC m=+726.914865610" watchObservedRunningTime="2026-07-09 18:15:04.828844696 +0000 UTC m=+726.916094017"
Jul 09 18:15:13.806404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:13.806375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-jpp7h"
Jul 09 18:15:15.816594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:15:15.816565    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-chmc9"
Jul 09 18:16:50.056104 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.056063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:16:50.059720 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.059683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.062580 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.062546    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:16:50.062766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.062578    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:16:50.062766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.062603    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-mg5cs\""
Jul 09 18:16:50.063001 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.062983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:16:50.076390 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.076348    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:16:50.141912 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.141877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.141912 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.141913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.141936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndq4\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142190 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.142475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.142283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndq4\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243378 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243702 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243702 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243820 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243998 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.243998 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.243987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.244105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.244079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.246371 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.246348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.246736 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.246710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.255510 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.255459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.255698 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.255578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndq4\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4\") pod \"router-gateway-1-openshift-default-6679d9568-blxjb\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.337451 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.337350    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:16:50.342651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.342615    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.343870 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.343844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.343968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.343879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.343968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.343903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmv8\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.343968 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.343951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.344139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.343983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.344139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.344006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.344139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.344022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.344139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.344089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.344139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.344122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.357250 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.357195    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:16:50.371909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.371566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmv8\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.445452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.446152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.446430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.446698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.448976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.446919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.449949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.449710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.450621 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.450564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.456905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.456866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmv8\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.457174 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.457151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-29zcg\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.534086 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.534036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:16:50.536443 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:16:50.536414    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fce5426_7d7b_4c19_a102_9a5aebb10255.slice/crio-21e5435a4cc0f8c4c374b984c9ba1cd94ac83bd3cabcdeb2c181245a0a857c8c WatchSource:0}: Error finding container 21e5435a4cc0f8c4c374b984c9ba1cd94ac83bd3cabcdeb2c181245a0a857c8c: Status 404 returned error can't find the container with id 21e5435a4cc0f8c4c374b984c9ba1cd94ac83bd3cabcdeb2c181245a0a857c8c
Jul 09 18:16:50.538952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.538910    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:50.539058 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.539015    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:50.539101 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.539057    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:50.657686 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.657638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:50.832384 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.832353    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:16:50.835732 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:16:50.835694    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f39f21b_380e_4cc4_998b_0f06d6575933.slice/crio-aba3beff780cbae4154ba13472b5764a16b197e799821cab3c962134d6456f8e WatchSource:0}: Error finding container aba3beff780cbae4154ba13472b5764a16b197e799821cab3c962134d6456f8e: Status 404 returned error can't find the container with id aba3beff780cbae4154ba13472b5764a16b197e799821cab3c962134d6456f8e
Jul 09 18:16:50.838209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.838168    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:50.838372 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.838270    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:50.838372 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:50.838303    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:16:51.366270 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.366219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" event={"ID":"0fce5426-7d7b-4c19-a102-9a5aebb10255","Type":"ContainerStarted","Data":"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"}
Jul 09 18:16:51.366841 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.366327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" event={"ID":"0fce5426-7d7b-4c19-a102-9a5aebb10255","Type":"ContainerStarted","Data":"21e5435a4cc0f8c4c374b984c9ba1cd94ac83bd3cabcdeb2c181245a0a857c8c"}
Jul 09 18:16:51.368095 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.368065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" event={"ID":"4f39f21b-380e-4cc4-998b-0f06d6575933","Type":"ContainerStarted","Data":"4c4f56b3b9906c6a6ac2e83bc5136a5ec344730c6e2e0d1ef95f5ec812a3ea6b"}
Jul 09 18:16:51.368259 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.368125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" event={"ID":"4f39f21b-380e-4cc4-998b-0f06d6575933","Type":"ContainerStarted","Data":"aba3beff780cbae4154ba13472b5764a16b197e799821cab3c962134d6456f8e"}
Jul 09 18:16:51.371771 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.371739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:51.377582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.377551    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:51.391001 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.390928    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" podStartSLOduration=1.390902414 podStartE2EDuration="1.390902414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:51.38797576 +0000 UTC m=+833.475225090" watchObservedRunningTime="2026-07-09 18:16:51.390902414 +0000 UTC m=+833.478151724"
Jul 09 18:16:51.407527 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.407462    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" podStartSLOduration=1.407439332 podStartE2EDuration="1.407439332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:51.406941019 +0000 UTC m=+833.494190339" watchObservedRunningTime="2026-07-09 18:16:51.407439332 +0000 UTC m=+833.494688638"
Jul 09 18:16:51.658456 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.658368    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:51.663704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:51.663677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:52.373415 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:52.373378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:52.373870 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:52.373570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:52.374442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:52.374418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:16:52.374559 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:52.374545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:52.448684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:52.448652    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:16:54.385853 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:54.385746    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" podUID="0fce5426-7d7b-4c19-a102-9a5aebb10255" containerName="istio-proxy" containerID="cri-o://44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039" gracePeriod=30
Jul 09 18:16:59.639772 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.639743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:16:59.728376 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728444    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728463    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728544    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728571    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728609    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fndq4\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4\") pod \"0fce5426-7d7b-4c19-a102-9a5aebb10255\" (UID: \"0fce5426-7d7b-4c19-a102-9a5aebb10255\") "
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728680    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728909    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:59.728935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728927    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.729155 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.728882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:59.729354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.729320    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:59.730959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.730931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:16:59.731069 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.730975    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4" (OuterVolumeSpecName: "kube-api-access-fndq4") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "kube-api-access-fndq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:59.731145 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.731125    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:59.731193 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.731151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fce5426-7d7b-4c19-a102-9a5aebb10255" (UID: "0fce5426-7d7b-4c19-a102-9a5aebb10255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:59.830347 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830309    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830347 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830343    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fce5426-7d7b-4c19-a102-9a5aebb10255-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830359    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830371    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830385    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830397    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fce5426-7d7b-4c19-a102-9a5aebb10255-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:16:59.830566 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:16:59.830409    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndq4\" (UniqueName: \"kubernetes.io/projected/0fce5426-7d7b-4c19-a102-9a5aebb10255-kube-api-access-fndq4\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:00.418946 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.418906    2565 generic.go:356] "Generic (PLEG): container finished" podID="0fce5426-7d7b-4c19-a102-9a5aebb10255" containerID="44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039" exitCode=0
Jul 09 18:17:00.419123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.418954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"
Jul 09 18:17:00.419123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.418992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" event={"ID":"0fce5426-7d7b-4c19-a102-9a5aebb10255","Type":"ContainerDied","Data":"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"}
Jul 09 18:17:00.419123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.419061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb" event={"ID":"0fce5426-7d7b-4c19-a102-9a5aebb10255","Type":"ContainerDied","Data":"21e5435a4cc0f8c4c374b984c9ba1cd94ac83bd3cabcdeb2c181245a0a857c8c"}
Jul 09 18:17:00.419123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.419090    2565 scope.go:117] "RemoveContainer" containerID="44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"
Jul 09 18:17:00.429154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.429137    2565 scope.go:117] "RemoveContainer" containerID="44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"
Jul 09 18:17:00.429457 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:17:00.429437    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039\": container with ID starting with 44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039 not found: ID does not exist" containerID="44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"
Jul 09 18:17:00.429514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.429471    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039"} err="failed to get container status \"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039\": rpc error: code = NotFound desc = could not find container \"44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039\": container with ID starting with 44c2cb5eec68b5a47f21627f20ae8f01b62f335b62dd9bc59059d0e7ab45c039 not found: ID does not exist"
Jul 09 18:17:00.441733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.441707    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:17:00.445214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.445191    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-blxjb"]
Jul 09 18:17:00.481879 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:00.481851    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fce5426-7d7b-4c19-a102-9a5aebb10255" path="/var/lib/kubelet/pods/0fce5426-7d7b-4c19-a102-9a5aebb10255/volumes"
Jul 09 18:17:03.077683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.077639    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:17:03.078185 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.078128    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fce5426-7d7b-4c19-a102-9a5aebb10255" containerName="istio-proxy"
Jul 09 18:17:03.078185 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.078144    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fce5426-7d7b-4c19-a102-9a5aebb10255" containerName="istio-proxy"
Jul 09 18:17:03.078307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.078223    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fce5426-7d7b-4c19-a102-9a5aebb10255" containerName="istio-proxy"
Jul 09 18:17:03.081497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.081477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.093147 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.093119    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:17:03.160796 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.160796 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.160977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.161010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.161071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkz7l\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.161188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.161119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262011 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.261970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262213 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkz7l\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262592 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262888 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.262965 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.262944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.264759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.264736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.264907 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.264888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.271047 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.271016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.271273 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.271225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkz7l\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l\") pod \"router-gateway-1-openshift-default-6679d9568-fn4fv\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.392564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.392473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:03.593908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.593887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:17:03.595947 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:17:03.595917    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96346bc_f179_4e5d_86e6_44dcfb19a766.slice/crio-b8b9b434b030f22049879efb8e077ff56e112b0b81d8be37933d6c4338ebc3a4 WatchSource:0}: Error finding container b8b9b434b030f22049879efb8e077ff56e112b0b81d8be37933d6c4338ebc3a4: Status 404 returned error can't find the container with id b8b9b434b030f22049879efb8e077ff56e112b0b81d8be37933d6c4338ebc3a4
Jul 09 18:17:03.598047 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.598010    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:03.598121 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.598080    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:03.598121 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:03.598113    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:04.453534 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:04.453489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerStarted","Data":"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"}
Jul 09 18:17:04.454028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:04.453561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerStarted","Data":"b8b9b434b030f22049879efb8e077ff56e112b0b81d8be37933d6c4338ebc3a4"}
Jul 09 18:17:04.496294 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:04.496242    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podStartSLOduration=1.496213799 podStartE2EDuration="1.496213799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:04.494794887 +0000 UTC m=+846.582044173" watchObservedRunningTime="2026-07-09 18:17:04.496213799 +0000 UTC m=+846.583463103"
Jul 09 18:17:05.393318 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.393281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:05.398335 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.398308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:05.458598 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.458559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:05.459823 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.459801    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:05.541533 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.541499    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:17:05.541969 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:05.541939    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerName="istio-proxy" containerID="cri-o://4c4f56b3b9906c6a6ac2e83bc5136a5ec344730c6e2e0d1ef95f5ec812a3ea6b" gracePeriod=30
Jul 09 18:17:07.415345 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:07.415279    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:11.498163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.498131    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerID="4c4f56b3b9906c6a6ac2e83bc5136a5ec344730c6e2e0d1ef95f5ec812a3ea6b" exitCode=0
Jul 09 18:17:11.498590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.498202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" event={"ID":"4f39f21b-380e-4cc4-998b-0f06d6575933","Type":"ContainerDied","Data":"4c4f56b3b9906c6a6ac2e83bc5136a5ec344730c6e2e0d1ef95f5ec812a3ea6b"}
Jul 09 18:17:11.636260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.632351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:17:11.756748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756694    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.756748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.756748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756764    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.756986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757097    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gmv8\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757173    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo\") pod \"4f39f21b-380e-4cc4-998b-0f06d6575933\" (UID: \"4f39f21b-380e-4cc4-998b-0f06d6575933\") "
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757222    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:11.757370 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:11.757670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757537    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.757670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757552    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.757670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757571    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39f21b-380e-4cc4-998b-0f06d6575933-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.757670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757580    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.757670 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.757588    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.759706 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.759676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:17:11.759792 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.759699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:11.759792 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.759753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8" (OuterVolumeSpecName: "kube-api-access-2gmv8") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "kube-api-access-2gmv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:11.759904 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.759812    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f39f21b-380e-4cc4-998b-0f06d6575933" (UID: "4f39f21b-380e-4cc4-998b-0f06d6575933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:11.858407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.858371    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.858407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.858401    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.858407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.858415    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gmv8\" (UniqueName: \"kubernetes.io/projected/4f39f21b-380e-4cc4-998b-0f06d6575933-kube-api-access-2gmv8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:11.858680 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:11.858429    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39f21b-380e-4cc4-998b-0f06d6575933-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:17:12.506473 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:12.506433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg" event={"ID":"4f39f21b-380e-4cc4-998b-0f06d6575933","Type":"ContainerDied","Data":"aba3beff780cbae4154ba13472b5764a16b197e799821cab3c962134d6456f8e"}
Jul 09 18:17:12.506915 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:12.506492    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"
Jul 09 18:17:12.506915 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:12.506509    2565 scope.go:117] "RemoveContainer" containerID="4c4f56b3b9906c6a6ac2e83bc5136a5ec344730c6e2e0d1ef95f5ec812a3ea6b"
Jul 09 18:17:12.536001 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:12.535958    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:17:12.541560 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:12.541514    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-29zcg"]
Jul 09 18:17:14.481967 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:14.481924    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" path="/var/lib/kubelet/pods/4f39f21b-380e-4cc4-998b-0f06d6575933/volumes"
Jul 09 18:17:16.531740 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.531703    2565 generic.go:356] "Generic (PLEG): container finished" podID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerID="5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a" exitCode=0
Jul 09 18:17:16.532176 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.531771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerDied","Data":"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"}
Jul 09 18:17:16.532176 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.532169    2565 scope.go:117] "RemoveContainer" containerID="5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"
Jul 09 18:17:16.533253 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.533210    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:16.533326 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.533314    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:16.533380 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:16.533363    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:17.542629 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:17.542585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerStarted","Data":"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"}
Jul 09 18:17:18.393020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:18.392986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:18.394341 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:18.394312    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:17:19.393035 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:19.392995    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:17:20.393613 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:20.393575    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:17:20.458961 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:20.458903    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:21.393355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:21.393314    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:17:22.393087 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:22.393037    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:17:23.397299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:23.397268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:23.398327 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:23.398309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:17:29.196508 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.196469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:17:29.196910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.196844    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerName="istio-proxy"
Jul 09 18:17:29.196910 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.196855    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerName="istio-proxy"
Jul 09 18:17:29.196991 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.196925    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f39f21b-380e-4cc4-998b-0f06d6575933" containerName="istio-proxy"
Jul 09 18:17:29.200142 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.200121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.203090 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.203066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:17:29.203260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.203076    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:17:29.203361 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.203268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 18:17:29.216012 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.215989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:17:29.322704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8x8\" (UniqueName: \"kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.322980 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.322934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424249 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8x8\" (UniqueName: \"kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424854 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.424925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.424905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.426706 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.426685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.426990 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.426969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.437850 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.437824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8x8\" (UniqueName: \"kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-fvh94\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.511209 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.511124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:29.659543 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.659509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:17:29.662086 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:17:29.662044    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfbb3eb_28f8_4dbf_9db8_2696fe7a5270.slice/crio-cd4aa302d79c61651eb83df675f7a3223d01cd78f23f3e79dd70645c8454a079 WatchSource:0}: Error finding container cd4aa302d79c61651eb83df675f7a3223d01cd78f23f3e79dd70645c8454a079: Status 404 returned error can't find the container with id cd4aa302d79c61651eb83df675f7a3223d01cd78f23f3e79dd70645c8454a079
Jul 09 18:17:29.664342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.664271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:17:29.669129 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.669110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.671479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.671460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-8sgqh\""
Jul 09 18:17:29.680113 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.680088    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:17:29.828188 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.828369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.828419 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.828455 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.828455 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxp8\" (UniqueName: \"kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.828521 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.828473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.928903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.928865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.928903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.928900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.928920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.928980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929173 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxp8\" (UniqueName: \"kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929509 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.929509 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.929478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.931586 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.931567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.937081 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.937061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxp8\" (UniqueName: \"kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:29.980903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:29.980873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:17:30.141392 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:30.141366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:17:30.143092 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:17:30.143064    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfc0cc7_a7e0_4de3_a17a_cca6d21aadde.slice/crio-6358de9f76bc2cad48a016237966fc68352554f242671215d01d1c10973e9a59 WatchSource:0}: Error finding container 6358de9f76bc2cad48a016237966fc68352554f242671215d01d1c10973e9a59: Status 404 returned error can't find the container with id 6358de9f76bc2cad48a016237966fc68352554f242671215d01d1c10973e9a59
Jul 09 18:17:30.625386 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:30.625337    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerStarted","Data":"6358de9f76bc2cad48a016237966fc68352554f242671215d01d1c10973e9a59"}
Jul 09 18:17:30.626863 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:30.626834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerStarted","Data":"cd4aa302d79c61651eb83df675f7a3223d01cd78f23f3e79dd70645c8454a079"}
Jul 09 18:17:34.670497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:34.670391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerStarted","Data":"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"}
Jul 09 18:17:34.673033 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:34.673001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerStarted","Data":"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"}
Jul 09 18:17:35.680893 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:35.680857    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerID="df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb" exitCode=0
Jul 09 18:17:35.681299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:35.680952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerDied","Data":"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"}
Jul 09 18:17:37.700497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:37.700454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerStarted","Data":"ef204bc52c8796c3386d8784abc7ed04856a36950b2b5bf11617cacb1c789d29"}
Jul 09 18:17:38.709795 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:38.709752    2565 generic.go:356] "Generic (PLEG): container finished" podID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerID="04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8" exitCode=0
Jul 09 18:17:38.710216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:38.709837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerDied","Data":"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"}
Jul 09 18:17:40.656773 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.656438    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:58100->10.132.0.36:15021: read: connection reset by peer" start-of-body=
Jul 09 18:17:40.656773 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.656512    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:58100->10.132.0.36:15021: read: connection reset by peer"
Jul 09 18:17:40.729651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.729607    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerID="45d19395c85236eb3e2cdd5c703fb6bace879b39461eb68d70a9d0e7f90d4994" exitCode=0
Jul 09 18:17:40.729854 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.729684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" event={"ID":"ff85a442-d24b-4a72-a758-81e518cfd1fd","Type":"ContainerDied","Data":"45d19395c85236eb3e2cdd5c703fb6bace879b39461eb68d70a9d0e7f90d4994"}
Jul 09 18:17:40.730273 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.730226    2565 scope.go:117] "RemoveContainer" containerID="45d19395c85236eb3e2cdd5c703fb6bace879b39461eb68d70a9d0e7f90d4994"
Jul 09 18:17:40.731594 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.731554    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:40.731719 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.731646    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:40.731719 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.731700    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:17:40.732154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.732132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerStarted","Data":"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"}
Jul 09 18:17:40.790748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:40.790628    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" podStartSLOduration=1.8149189730000002 podStartE2EDuration="11.790602595s" totalImagesPullingTime="6.436892908s" totalInitContainerRuntime="3.538790714s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:40.786932152 +0000 UTC m=+882.874181470" watchObservedRunningTime="2026-07-09 18:17:40.790602595 +0000 UTC m=+882.877851900"
Jul 09 18:17:41.512124 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:41.512088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:41.749876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:41.749836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" event={"ID":"ff85a442-d24b-4a72-a758-81e518cfd1fd","Type":"ContainerStarted","Data":"7153a14b8b9e398b535573b0b962b86a0a5b18892b63836ac79ff378eab31a9f"}
Jul 09 18:17:42.711019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:42.710964    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:17:42.711371 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:42.711342    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 09 18:17:42.711457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:42.711423    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 09 18:17:43.711026 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:43.710992    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 09 18:17:43.711512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:43.711053    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 09 18:17:44.711681 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:44.711441    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 09 18:17:44.711681 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:44.711526    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 09 18:17:45.711615 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:45.711578    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 09 18:17:45.711809 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:45.711650    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 09 18:17:46.710805 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:46.710724    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 09 18:17:46.710805 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:46.710801    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb" podUID="ff85a442-d24b-4a72-a758-81e518cfd1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 09 18:17:47.794702 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.794659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:17:47.795288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.795224    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerID="ef204bc52c8796c3386d8784abc7ed04856a36950b2b5bf11617cacb1c789d29" exitCode=1
Jul 09 18:17:47.795438 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.795317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerDied","Data":"ef204bc52c8796c3386d8784abc7ed04856a36950b2b5bf11617cacb1c789d29"}
Jul 09 18:17:47.878979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.878837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:17:47.879345 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.879324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:17:47.881652 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:47.881594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-wqvjb"
Jul 09 18:17:49.178908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:49.178852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:17:49.179387 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:49.179139    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" containerID="cri-o://4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d" gracePeriod=30
Jul 09 18:17:49.511490 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:49.511448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:49.531215 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:49.531188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:49.825817 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:49.825723    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:17:53.400891 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:17:53.400838    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:08.399205 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:08.399156    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Jul 09 18:18:09.284436 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.284401    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:18:09.442183 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkz7l\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l\") pod \"d96346bc-f179-4e5d-86e6-44dcfb19a766\" (UID: \"d96346bc-f179-4e5d-86e6-44dcfb19a766\") "
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data" (OuterVolumeSpecName: "istio-data") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442601    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.442640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442912    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442932    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442946    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442956    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.443154 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.442952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:09.444631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.444610    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:18:09.444985 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.444958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.445120 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.445049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token" (OuterVolumeSpecName: "istio-token") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:09.445120 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.445093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l" (OuterVolumeSpecName: "kube-api-access-nkz7l") pod "d96346bc-f179-4e5d-86e6-44dcfb19a766" (UID: "d96346bc-f179-4e5d-86e6-44dcfb19a766"). InnerVolumeSpecName "kube-api-access-nkz7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:09.544140 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.544103    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkz7l\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-kube-api-access-nkz7l\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.544140 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.544139    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.544140 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.544152    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.544453 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.544163    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96346bc-f179-4e5d-86e6-44dcfb19a766-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.544453 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.544174    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96346bc-f179-4e5d-86e6-44dcfb19a766-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.670852 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.670825    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:18:09.671024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.670919    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:18:09.685280 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.685256    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:18:09.685462 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.685306    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:18:09.689817 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.689795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:18:09.689941 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.689909    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:18:09.696728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.696672    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:18:09.697061 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.697045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:18:09.942850 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.942817    2565 generic.go:356] "Generic (PLEG): container finished" podID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerID="4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d" exitCode=0
Jul 09 18:18:09.943071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.942890    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"
Jul 09 18:18:09.943071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.942907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerDied","Data":"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"}
Jul 09 18:18:09.943071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.942980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv" event={"ID":"d96346bc-f179-4e5d-86e6-44dcfb19a766","Type":"ContainerDied","Data":"b8b9b434b030f22049879efb8e077ff56e112b0b81d8be37933d6c4338ebc3a4"}
Jul 09 18:18:09.943071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.943016    2565 scope.go:117] "RemoveContainer" containerID="4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"
Jul 09 18:18:09.945168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.945150    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:18:09.945656 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.945629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerStarted","Data":"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"}
Jul 09 18:18:09.946017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.945999    2565 scope.go:117] "RemoveContainer" containerID="ef204bc52c8796c3386d8784abc7ed04856a36950b2b5bf11617cacb1c789d29"
Jul 09 18:18:09.953407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.953317    2565 scope.go:117] "RemoveContainer" containerID="5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"
Jul 09 18:18:09.966881 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.966852    2565 scope.go:117] "RemoveContainer" containerID="4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"
Jul 09 18:18:09.967260 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:09.967216    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d\": container with ID starting with 4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d not found: ID does not exist" containerID="4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"
Jul 09 18:18:09.967393 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.967273    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d"} err="failed to get container status \"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d\": rpc error: code = NotFound desc = could not find container \"4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d\": container with ID starting with 4c9fe4ac6971303fe5d0d5048f34714316b31176da531c7ddc63f30f9964994d not found: ID does not exist"
Jul 09 18:18:09.967393 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.967304    2565 scope.go:117] "RemoveContainer" containerID="5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"
Jul 09 18:18:09.967616 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:09.967598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a\": container with ID starting with 5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a not found: ID does not exist" containerID="5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"
Jul 09 18:18:09.967663 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.967625    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a"} err="failed to get container status \"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a\": rpc error: code = NotFound desc = could not find container \"5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a\": container with ID starting with 5ef6bb20c7d230b208919de017f4c0c2ea5eaad550dc824f595d3fb4dfbead2a not found: ID does not exist"
Jul 09 18:18:09.981020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.980990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:09.981166 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.981051    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:09.981166 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.981071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:09.981166 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:09.981084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:10.005416 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.005377    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:18:10.005540 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.005438    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-fn4fv"]
Jul 09 18:18:10.482705 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.482670    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" path="/var/lib/kubelet/pods/d96346bc-f179-4e5d-86e6-44dcfb19a766/volumes"
Jul 09 18:18:10.953025 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.952995    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:18:10.953541 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.953509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerStarted","Data":"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"}
Jul 09 18:18:10.953697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.953635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:10.978751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:10.978688    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podStartSLOduration=3.117558638 podStartE2EDuration="41.97866662s" totalImagesPullingTime="38.07078497s" totalInitContainerRuntime="790.323012ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:10.977169194 +0000 UTC m=+913.064418498" watchObservedRunningTime="2026-07-09 18:18:10.97866662 +0000 UTC m=+913.065915925"
Jul 09 18:18:19.982641 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:18:19.982607    2565 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.50:9003", ServerName: "10.132.0.50:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.50:9003: connect: connection refused"
Jul 09 18:18:19.984019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:19.983993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:19.985294 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:19.985268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:20.982292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:20.982246    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.50:9003\" within 1s: context deadline exceeded"
Jul 09 18:18:29.982469 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:18:29.982436    2565 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.50:9003", ServerName: "10.132.0.50:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.50:9003: connect: connection refused"
Jul 09 18:18:30.982443 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:30.982398    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.50:9003\" within 1s: context deadline exceeded"
Jul 09 18:18:40.017433 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:40.017399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:42.274109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:42.274066    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:18:42.274634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:42.274389    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="tokenizer" containerID="cri-o://442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49" gracePeriod=60
Jul 09 18:18:42.277497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:42.277468    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:18:42.285153 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:42.285118    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:18:42.285153 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:42.285153 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:42.285153 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073" cmd=["/bin/sleep","15"]
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:42.285172    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" message="" err=<
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:42.285185    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main" containerID="cri-o://b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"
Jul 09 18:18:42.285554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:42.285348    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main" containerID="cri-o://b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073" gracePeriod=60
Jul 09 18:18:43.150392 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.150363    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s_8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/main/0.log"
Jul 09 18:18:43.150766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.150742    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerID="b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073" exitCode=0
Jul 09 18:18:43.150856 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.150819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerDied","Data":"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"}
Jul 09 18:18:43.150918 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.150900    2565 scope.go:117] "RemoveContainer" containerID="ef204bc52c8796c3386d8784abc7ed04856a36950b2b5bf11617cacb1c789d29"
Jul 09 18:18:43.844730 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.844703    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:43.954284 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954461 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954329    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954461 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954461 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954404    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lxp8\" (UniqueName: \"kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954461 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954484    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache\") pod \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\" (UID: \"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde\") "
Jul 09 18:18:43.954684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:43.954684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:43.954858 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954782    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-tmp\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:43.954858 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954797    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:43.954927 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.954857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:43.955172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.955144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:43.956589 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.956564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:43.956683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:43.956580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8" (OuterVolumeSpecName: "kube-api-access-5lxp8") pod "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" (UID: "8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde"). InnerVolumeSpecName "kube-api-access-5lxp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:44.055925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.055890    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:44.055925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.055919    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxp8\" (UniqueName: \"kubernetes.io/projected/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kube-api-access-5lxp8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:44.055925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.055928    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:44.056163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.055939    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde-tokenizer-cache\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:44.158486 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.158450    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerID="442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49" exitCode=0
Jul 09 18:18:44.158654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.158516    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"
Jul 09 18:18:44.158654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.158532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerDied","Data":"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"}
Jul 09 18:18:44.158654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.158587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s" event={"ID":"8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde","Type":"ContainerDied","Data":"6358de9f76bc2cad48a016237966fc68352554f242671215d01d1c10973e9a59"}
Jul 09 18:18:44.158654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.158610    2565 scope.go:117] "RemoveContainer" containerID="b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"
Jul 09 18:18:44.168201 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.168185    2565 scope.go:117] "RemoveContainer" containerID="442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"
Jul 09 18:18:44.176480 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.176460    2565 scope.go:117] "RemoveContainer" containerID="df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"
Jul 09 18:18:44.184993 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.184967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:18:44.188684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.188627    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7744d6c72dt6s"]
Jul 09 18:18:44.188684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.188648    2565 scope.go:117] "RemoveContainer" containerID="b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"
Jul 09 18:18:44.189283 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:44.189218    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073\": container with ID starting with b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073 not found: ID does not exist" containerID="b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"
Jul 09 18:18:44.189283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.189272    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073"} err="failed to get container status \"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073\": rpc error: code = NotFound desc = could not find container \"b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073\": container with ID starting with b2a181a8b63be4ad5fc940f0324e8e774b7a32c16f00fa17f2a0980386522073 not found: ID does not exist"
Jul 09 18:18:44.189382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.189292    2565 scope.go:117] "RemoveContainer" containerID="442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"
Jul 09 18:18:44.189844 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:44.189823    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49\": container with ID starting with 442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49 not found: ID does not exist" containerID="442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"
Jul 09 18:18:44.189914 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.189849    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49"} err="failed to get container status \"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49\": rpc error: code = NotFound desc = could not find container \"442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49\": container with ID starting with 442cd3ca9b7d3014825cfb487d87b3209511c2c53e083f2c9eaf2966eaf36b49 not found: ID does not exist"
Jul 09 18:18:44.189914 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.189865    2565 scope.go:117] "RemoveContainer" containerID="df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"
Jul 09 18:18:44.190104 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:44.190087    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb\": container with ID starting with df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb not found: ID does not exist" containerID="df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"
Jul 09 18:18:44.190161 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.190110    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb"} err="failed to get container status \"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb\": rpc error: code = NotFound desc = could not find container \"df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb\": container with ID starting with df2c88bd6bc7e967d7a6e1805e46870361bc46255742cf52dee5c97c8aa6d8fb not found: ID does not exist"
Jul 09 18:18:44.482677 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:44.482645    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" path="/var/lib/kubelet/pods/8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde/volumes"
Jul 09 18:18:57.300676 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.300636    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="main" containerID="cri-o://d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1" gracePeriod=45
Jul 09 18:18:57.560969 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.560907    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:18:57.676852 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.676824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.676852 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.676855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.676881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.676898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb8x8\" (UniqueName: \"kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.676992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs\") pod \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\" (UID: \"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270\") "
Jul 09 18:18:57.677329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:57.677329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home" (OuterVolumeSpecName: "home") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:57.677329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:57.677512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677409    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tmp-dir\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.677512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677424    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-home\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.677512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.677433    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-model-cache\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.679271 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.679225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm" (OuterVolumeSpecName: "dshm") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:57.679380 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.679266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:57.679417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.679406    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8" (OuterVolumeSpecName: "kube-api-access-cb8x8") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "kube-api-access-cb8x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:57.733738 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.733694    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" (UID: "7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:57.778063 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.778033    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.778063 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.778062    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8x8\" (UniqueName: \"kubernetes.io/projected/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kube-api-access-cb8x8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.778214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.778071    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-dshm\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:57.778214 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:57.778079    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.233512 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.233475    2565 generic.go:356] "Generic (PLEG): container finished" podID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerID="d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1" exitCode=0
Jul 09 18:18:58.233701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.233532    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"
Jul 09 18:18:58.233701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.233551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerDied","Data":"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"}
Jul 09 18:18:58.233701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.233604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94" event={"ID":"7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270","Type":"ContainerDied","Data":"cd4aa302d79c61651eb83df675f7a3223d01cd78f23f3e79dd70645c8454a079"}
Jul 09 18:18:58.233701 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.233631    2565 scope.go:117] "RemoveContainer" containerID="d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"
Jul 09 18:18:58.242824 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.242808    2565 scope.go:117] "RemoveContainer" containerID="04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"
Jul 09 18:18:58.250944 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.250925    2565 scope.go:117] "RemoveContainer" containerID="d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"
Jul 09 18:18:58.251215 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:58.251196    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1\": container with ID starting with d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1 not found: ID does not exist" containerID="d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"
Jul 09 18:18:58.251312 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.251224    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1"} err="failed to get container status \"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1\": rpc error: code = NotFound desc = could not find container \"d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1\": container with ID starting with d0e61313f9477718c596df4bf71a141aec040683fea0376eab34a1dcbca767b1 not found: ID does not exist"
Jul 09 18:18:58.251312 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.251268    2565 scope.go:117] "RemoveContainer" containerID="04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"
Jul 09 18:18:58.251511 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:18:58.251494    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8\": container with ID starting with 04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8 not found: ID does not exist" containerID="04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"
Jul 09 18:18:58.251553 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.251517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8"} err="failed to get container status \"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8\": rpc error: code = NotFound desc = could not find container \"04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8\": container with ID starting with 04a911031a81baec75cfd3948cf80594eb41ef7d38a88a320465df613f9b6cc8 not found: ID does not exist"
Jul 09 18:18:58.257993 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.257970    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:18:58.263034 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.263012    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-fvh94"]
Jul 09 18:18:58.482171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:18:58.482134    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" path="/var/lib/kubelet/pods/7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270/volumes"
Jul 09 18:19:18.295753 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.295710    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296155    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="storage-initializer"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296168    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="storage-initializer"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296177    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296182    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296189    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="tokenizer"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296194    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="tokenizer"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296201    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296205    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296213    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296218    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296252    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296260    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296275    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="storage-initializer"
Jul 09 18:19:18.296301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296283    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="storage-initializer"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296369    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfbb3eb-28f8-4dbf-9db8-2696fe7a5270" containerName="main"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296380    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="tokenizer"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296393    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296402    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296409    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfc0cc7-a7e0-4de3-a17a-cca6d21aadde" containerName="main"
Jul 09 18:19:18.297044 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.296422    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:19:18.301550 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.301530    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.304186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.304165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:19:18.305106 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.305084    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 09 18:19:18.305106 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.305093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-2cs6q\""
Jul 09 18:19:18.305424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.305093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:19:18.305424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.305164    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:19:18.309225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.309206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:18.349907 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.349876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhts\" (UniqueName: \"kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.349907 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.349920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.350113 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.349977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.450741 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.450706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.450922 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.450757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.450922 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.450819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhts\" (UniqueName: \"kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.451122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.451102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.453290 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.453264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.460507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.460480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhts\" (UniqueName: \"kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.612522 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.612424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:18.753469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:18.753403    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:18.755623 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:19:18.755595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb6c077_e50a_4733_97b9_73514972ec55.slice/crio-cf9dc063862b82ca29a09b6a3e20444c86ef6da03475cf84ec7445d11bf48920 WatchSource:0}: Error finding container cf9dc063862b82ca29a09b6a3e20444c86ef6da03475cf84ec7445d11bf48920: Status 404 returned error can't find the container with id cf9dc063862b82ca29a09b6a3e20444c86ef6da03475cf84ec7445d11bf48920
Jul 09 18:19:19.339355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:19.339299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" event={"ID":"0eb6c077-e50a-4733-97b9-73514972ec55","Type":"ContainerStarted","Data":"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"}
Jul 09 18:19:19.339355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:19.339368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" event={"ID":"0eb6c077-e50a-4733-97b9-73514972ec55","Type":"ContainerStarted","Data":"cf9dc063862b82ca29a09b6a3e20444c86ef6da03475cf84ec7445d11bf48920"}
Jul 09 18:19:19.339792 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:19.339387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:19.357518 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:19.357469    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" podStartSLOduration=1.357451845 podStartE2EDuration="1.357451845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:19.355883233 +0000 UTC m=+981.443132547" watchObservedRunningTime="2026-07-09 18:19:19.357451845 +0000 UTC m=+981.444701150"
Jul 09 18:19:50.346646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:50.346619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:51.518302 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.518208    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:51.527876 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:19:51.527841    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:19:51.527876 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:51.527876 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:51.527876 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b" cmd=["/bin/sleep","15"]
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.527893    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" message="" err=<
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:19:51.527905    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" containerName="main" containerID="cri-o://72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"
Jul 09 18:19:51.528206 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.527956    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" containerName="main" containerID="cri-o://72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b" gracePeriod=60
Jul 09 18:19:51.778634 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.778565    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:51.951716 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.951683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmhts\" (UniqueName: \"kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts\") pod \"0eb6c077-e50a-4733-97b9-73514972ec55\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") "
Jul 09 18:19:51.951908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.951744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds\") pod \"0eb6c077-e50a-4733-97b9-73514972ec55\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") "
Jul 09 18:19:51.951908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.951821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs\") pod \"0eb6c077-e50a-4733-97b9-73514972ec55\" (UID: \"0eb6c077-e50a-4733-97b9-73514972ec55\") "
Jul 09 18:19:51.952054 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.952027    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0eb6c077-e50a-4733-97b9-73514972ec55" (UID: "0eb6c077-e50a-4733-97b9-73514972ec55"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:51.953830 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.953806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eb6c077-e50a-4733-97b9-73514972ec55" (UID: "0eb6c077-e50a-4733-97b9-73514972ec55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:51.953830 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:51.953827    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts" (OuterVolumeSpecName: "kube-api-access-xmhts") pod "0eb6c077-e50a-4733-97b9-73514972ec55" (UID: "0eb6c077-e50a-4733-97b9-73514972ec55"). InnerVolumeSpecName "kube-api-access-xmhts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:52.053059 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.052979    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb6c077-e50a-4733-97b9-73514972ec55-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:19:52.053059 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.053005    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhts\" (UniqueName: \"kubernetes.io/projected/0eb6c077-e50a-4733-97b9-73514972ec55-kube-api-access-xmhts\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:19:52.053059 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.053015    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0eb6c077-e50a-4733-97b9-73514972ec55-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:19:52.512441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.512397    2565 generic.go:356] "Generic (PLEG): container finished" podID="0eb6c077-e50a-4733-97b9-73514972ec55" containerID="72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b" exitCode=0
Jul 09 18:19:52.512441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.512442    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"
Jul 09 18:19:52.512691 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.512484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" event={"ID":"0eb6c077-e50a-4733-97b9-73514972ec55","Type":"ContainerDied","Data":"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"}
Jul 09 18:19:52.512691 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.512557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5" event={"ID":"0eb6c077-e50a-4733-97b9-73514972ec55","Type":"ContainerDied","Data":"cf9dc063862b82ca29a09b6a3e20444c86ef6da03475cf84ec7445d11bf48920"}
Jul 09 18:19:52.512691 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.512584    2565 scope.go:117] "RemoveContainer" containerID="72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"
Jul 09 18:19:52.521919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.521898    2565 scope.go:117] "RemoveContainer" containerID="72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"
Jul 09 18:19:52.522255 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:19:52.522214    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b\": container with ID starting with 72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b not found: ID does not exist" containerID="72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"
Jul 09 18:19:52.522306 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.522261    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b"} err="failed to get container status \"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b\": rpc error: code = NotFound desc = could not find container \"72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b\": container with ID starting with 72af2679d93de6c7044ab58e5d7d81b43f97ce1917ae7aa80fe27cd1e284781b not found: ID does not exist"
Jul 09 18:19:52.529198 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.529171    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:52.534879 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:52.534854    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheh7tf5"]
Jul 09 18:19:54.482191 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:19:54.482155    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" path="/var/lib/kubelet/pods/0eb6c077-e50a-4733-97b9-73514972ec55/volumes"
Jul 09 18:20:35.411187 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411119    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:20:35.411804 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411782    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" containerName="main"
Jul 09 18:20:35.411849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" containerName="main"
Jul 09 18:20:35.411849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411826    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:20:35.411849 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411835    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96346bc-f179-4e5d-86e6-44dcfb19a766" containerName="istio-proxy"
Jul 09 18:20:35.411956 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.411944    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb6c077-e50a-4733-97b9-73514972ec55" containerName="main"
Jul 09 18:20:35.415417 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.415395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.419130 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.419106    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:20:35.419314 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.419196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:20:35.419314 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.419196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-ksgt5\""
Jul 09 18:20:35.419464 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.419379    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:20:35.419521 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.419464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 09 18:20:35.423108 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.423087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:20:35.538200 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.538171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.538394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.538221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.538394 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.538280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgv7k\" (UniqueName: \"kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.638769 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.638729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.638975 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.638795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.638975 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.638820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgv7k\" (UniqueName: \"kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.639222 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.639194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.641302 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.641285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.647546 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.647521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgv7k\" (UniqueName: \"kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.727427 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.727393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:35.866553 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:35.866529    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:20:35.868797 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:20:35.868770    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9321e3a8_4cbf_4a83_945d_2ae273643bad.slice/crio-66ccdbdf186f5bdb99a2ea92c3e693c9fbfe5b944b721ba891b21b0690c7f9d6 WatchSource:0}: Error finding container 66ccdbdf186f5bdb99a2ea92c3e693c9fbfe5b944b721ba891b21b0690c7f9d6: Status 404 returned error can't find the container with id 66ccdbdf186f5bdb99a2ea92c3e693c9fbfe5b944b721ba891b21b0690c7f9d6
Jul 09 18:20:36.733802 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:36.733768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" event={"ID":"9321e3a8-4cbf-4a83-945d-2ae273643bad","Type":"ContainerStarted","Data":"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"}
Jul 09 18:20:36.734201 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:36.733822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" event={"ID":"9321e3a8-4cbf-4a83-945d-2ae273643bad","Type":"ContainerStarted","Data":"66ccdbdf186f5bdb99a2ea92c3e693c9fbfe5b944b721ba891b21b0690c7f9d6"}
Jul 09 18:20:36.734201 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:36.733883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:20:36.750879 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:20:36.750831    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" podStartSLOduration=1.750802469 podStartE2EDuration="1.750802469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:36.749398876 +0000 UTC m=+1058.836648181" watchObservedRunningTime="2026-07-09 18:20:36.750802469 +0000 UTC m=+1058.838051773"
Jul 09 18:21:07.741068 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:07.741030    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:21:24.242741 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.242701    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:21:24.246080 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.246057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.249775 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.249752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-sw7lj\""
Jul 09 18:21:24.249907 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.249791    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 09 18:21:24.261745 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.261719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:21:24.369872 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.369836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.370076 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.369884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hfn\" (UniqueName: \"kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.370076 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.369980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.470869 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.470815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.471075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.470891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hfn\" (UniqueName: \"kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.471075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.470941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.471436 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.471414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.473342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.473324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.481671 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.481640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hfn\" (UniqueName: \"kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.555674 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.555548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.705697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.705665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:21:24.707345 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:21:24.707318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367b0c62_0327_427c_9fc0_5d73649c1b0d.slice/crio-0f2c55815d2f91f0aee27615084fb1672c336d50c9198365617f358b4577c27e WatchSource:0}: Error finding container 0f2c55815d2f91f0aee27615084fb1672c336d50c9198365617f358b4577c27e: Status 404 returned error can't find the container with id 0f2c55815d2f91f0aee27615084fb1672c336d50c9198365617f358b4577c27e
Jul 09 18:21:24.997679 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.997638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" event={"ID":"367b0c62-0327-427c-9fc0-5d73649c1b0d","Type":"ContainerStarted","Data":"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"}
Jul 09 18:21:24.997915 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.997709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:21:24.997915 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:24.997724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" event={"ID":"367b0c62-0327-427c-9fc0-5d73649c1b0d","Type":"ContainerStarted","Data":"0f2c55815d2f91f0aee27615084fb1672c336d50c9198365617f358b4577c27e"}
Jul 09 18:21:25.016831 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:25.016762    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" podStartSLOduration=1.016741632 podStartE2EDuration="1.016741632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:25.015450887 +0000 UTC m=+1107.102700189" watchObservedRunningTime="2026-07-09 18:21:25.016741632 +0000 UTC m=+1107.103990937"
Jul 09 18:21:56.005987 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:21:56.005956    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:22:24.720057 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:24.719931    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:22:24.729874 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:24.729836    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:22:24.729874 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:24.729874 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:24.729874 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205" cmd=["/bin/sleep","15"]
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:24.729895    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" message="" err=<
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:24.729912    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerName="main" containerID="cri-o://c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"
Jul 09 18:22:24.730261 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:24.729956    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerName="main" containerID="cri-o://c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205" gracePeriod=60
Jul 09 18:22:24.980908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:24.980848    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:22:25.076990 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.076957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds\") pod \"367b0c62-0327-427c-9fc0-5d73649c1b0d\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") "
Jul 09 18:22:25.077191 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.077038    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2hfn\" (UniqueName: \"kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn\") pod \"367b0c62-0327-427c-9fc0-5d73649c1b0d\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") "
Jul 09 18:22:25.077191 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.077071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs\") pod \"367b0c62-0327-427c-9fc0-5d73649c1b0d\" (UID: \"367b0c62-0327-427c-9fc0-5d73649c1b0d\") "
Jul 09 18:22:25.077350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.077297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "367b0c62-0327-427c-9fc0-5d73649c1b0d" (UID: "367b0c62-0327-427c-9fc0-5d73649c1b0d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:25.079208 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.079175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "367b0c62-0327-427c-9fc0-5d73649c1b0d" (UID: "367b0c62-0327-427c-9fc0-5d73649c1b0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:25.079208 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.079184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn" (OuterVolumeSpecName: "kube-api-access-x2hfn") pod "367b0c62-0327-427c-9fc0-5d73649c1b0d" (UID: "367b0c62-0327-427c-9fc0-5d73649c1b0d"). InnerVolumeSpecName "kube-api-access-x2hfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:25.178398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.178365    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/367b0c62-0327-427c-9fc0-5d73649c1b0d-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:25.178398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.178393    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hfn\" (UniqueName: \"kubernetes.io/projected/367b0c62-0327-427c-9fc0-5d73649c1b0d-kube-api-access-x2hfn\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:25.178398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.178402    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367b0c62-0327-427c-9fc0-5d73649c1b0d-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:25.314352 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.314264    2565 generic.go:356] "Generic (PLEG): container finished" podID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerID="c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205" exitCode=0
Jul 09 18:22:25.314507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.314378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" event={"ID":"367b0c62-0327-427c-9fc0-5d73649c1b0d","Type":"ContainerDied","Data":"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"}
Jul 09 18:22:25.314507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.314421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk" event={"ID":"367b0c62-0327-427c-9fc0-5d73649c1b0d","Type":"ContainerDied","Data":"0f2c55815d2f91f0aee27615084fb1672c336d50c9198365617f358b4577c27e"}
Jul 09 18:22:25.314507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.314442    2565 scope.go:117] "RemoveContainer" containerID="c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"
Jul 09 18:22:25.314507 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.314381    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"
Jul 09 18:22:25.325127 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.325108    2565 scope.go:117] "RemoveContainer" containerID="c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"
Jul 09 18:22:25.325511 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:25.325490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205\": container with ID starting with c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205 not found: ID does not exist" containerID="c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"
Jul 09 18:22:25.325615 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.325517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205"} err="failed to get container status \"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205\": rpc error: code = NotFound desc = could not find container \"c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205\": container with ID starting with c825717471e7d20ff941834aa38d9725e8ee40bc5b6230efed5810facae0f205 not found: ID does not exist"
Jul 09 18:22:25.343361 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.343331    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:22:25.345318 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:25.345299    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche2ltdk"]
Jul 09 18:22:26.483864 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:26.483828    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" path="/var/lib/kubelet/pods/367b0c62-0327-427c-9fc0-5d73649c1b0d/volumes"
Jul 09 18:22:42.327511 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.327476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:22:42.337193 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:42.337145    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:22:42.337193 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:42.337193 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:42.337193 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9" cmd=["/bin/sleep","15"]
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.337219    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" message="" err=<
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:42.337249    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerName="main" containerID="cri-o://4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"
Jul 09 18:22:42.337474 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.337300    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerName="main" containerID="cri-o://4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9" gracePeriod=60
Jul 09 18:22:42.603062 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.603036    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:22:42.738660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.738626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs\") pod \"9321e3a8-4cbf-4a83-945d-2ae273643bad\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") "
Jul 09 18:22:42.738858 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.738725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds\") pod \"9321e3a8-4cbf-4a83-945d-2ae273643bad\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") "
Jul 09 18:22:42.738858 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.738755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgv7k\" (UniqueName: \"kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k\") pod \"9321e3a8-4cbf-4a83-945d-2ae273643bad\" (UID: \"9321e3a8-4cbf-4a83-945d-2ae273643bad\") "
Jul 09 18:22:42.739021 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.738991    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9321e3a8-4cbf-4a83-945d-2ae273643bad" (UID: "9321e3a8-4cbf-4a83-945d-2ae273643bad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:42.740921 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.740892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9321e3a8-4cbf-4a83-945d-2ae273643bad" (UID: "9321e3a8-4cbf-4a83-945d-2ae273643bad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:42.741028 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.740894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k" (OuterVolumeSpecName: "kube-api-access-bgv7k") pod "9321e3a8-4cbf-4a83-945d-2ae273643bad" (UID: "9321e3a8-4cbf-4a83-945d-2ae273643bad"). InnerVolumeSpecName "kube-api-access-bgv7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:42.839761 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.839674    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9321e3a8-4cbf-4a83-945d-2ae273643bad-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:42.839761 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.839716    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9321e3a8-4cbf-4a83-945d-2ae273643bad-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:42.839761 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:42.839730    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgv7k\" (UniqueName: \"kubernetes.io/projected/9321e3a8-4cbf-4a83-945d-2ae273643bad-kube-api-access-bgv7k\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:22:43.408939 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.408901    2565 generic.go:356] "Generic (PLEG): container finished" podID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerID="4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9" exitCode=0
Jul 09 18:22:43.408939 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.408939    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"
Jul 09 18:22:43.409442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.408993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" event={"ID":"9321e3a8-4cbf-4a83-945d-2ae273643bad","Type":"ContainerDied","Data":"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"}
Jul 09 18:22:43.409442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.409040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm" event={"ID":"9321e3a8-4cbf-4a83-945d-2ae273643bad","Type":"ContainerDied","Data":"66ccdbdf186f5bdb99a2ea92c3e693c9fbfe5b944b721ba891b21b0690c7f9d6"}
Jul 09 18:22:43.409442 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.409064    2565 scope.go:117] "RemoveContainer" containerID="4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"
Jul 09 18:22:43.419171 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.419151    2565 scope.go:117] "RemoveContainer" containerID="4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"
Jul 09 18:22:43.419466 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:22:43.419446    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9\": container with ID starting with 4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9 not found: ID does not exist" containerID="4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"
Jul 09 18:22:43.419554 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.419472    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9"} err="failed to get container status \"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9\": rpc error: code = NotFound desc = could not find container \"4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9\": container with ID starting with 4fd663d2cfa4653c551e353e7750a7002d0b812cb25687581ca340c99d3fbba9 not found: ID does not exist"
Jul 09 18:22:43.431585 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.431560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:22:43.435874 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:43.435853    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4x8qm"]
Jul 09 18:22:44.482851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:22:44.482810    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" path="/var/lib/kubelet/pods/9321e3a8-4cbf-4a83-945d-2ae273643bad/volumes"
Jul 09 18:23:09.720714 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.720683    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:23:09.721245 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.721212    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:23:09.725149 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.725131    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:23:09.725429 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.725413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:23:09.731709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.731683    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:23:09.731819 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:09.731804    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:23:22.189307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189273    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:23:22.189770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189668    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerName="main"
Jul 09 18:23:22.189770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189681    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerName="main"
Jul 09 18:23:22.189770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189705    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerName="main"
Jul 09 18:23:22.189770 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189713    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerName="main"
Jul 09 18:23:22.190002 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189795    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9321e3a8-4cbf-4a83-945d-2ae273643bad" containerName="main"
Jul 09 18:23:22.190002 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.189809    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="367b0c62-0327-427c-9fc0-5d73649c1b0d" containerName="main"
Jul 09 18:23:22.194504 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.194482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.197975 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.197907    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-6dwnm\""
Jul 09 18:23:22.197975 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.197934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:23:22.198168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.198007    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:23:22.198168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.198076    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:23:22.198485 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.198461    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 09 18:23:22.208123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.208099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:23:22.275441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.275403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.275617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.275476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vt9\" (UniqueName: \"kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.275617 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.275500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.376951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.376907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.377139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.376987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vt9\" (UniqueName: \"kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.377139 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.377014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.377391 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.377375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.379665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.379637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.386481 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.386454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vt9\" (UniqueName: \"kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.505485 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.505387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:22.657593 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:22.657561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:23:22.658177 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:23:22.658153    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb743270a_d647_4839_8938_35bde02d2463.slice/crio-d90cc22c7eb57f0517e4cf1de61d89f31e3227349acb151afe33a29fb474110f WatchSource:0}: Error finding container d90cc22c7eb57f0517e4cf1de61d89f31e3227349acb151afe33a29fb474110f: Status 404 returned error can't find the container with id d90cc22c7eb57f0517e4cf1de61d89f31e3227349acb151afe33a29fb474110f
Jul 09 18:23:23.620177 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:23.620144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" event={"ID":"b743270a-d647-4839-8938-35bde02d2463","Type":"ContainerStarted","Data":"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"}
Jul 09 18:23:23.620601 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:23.620227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" event={"ID":"b743270a-d647-4839-8938-35bde02d2463","Type":"ContainerStarted","Data":"d90cc22c7eb57f0517e4cf1de61d89f31e3227349acb151afe33a29fb474110f"}
Jul 09 18:23:23.620601 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:23.620267    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:23.641386 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:23.641334    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" podStartSLOduration=1.64131677 podStartE2EDuration="1.64131677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:23:23.639470109 +0000 UTC m=+1225.726719415" watchObservedRunningTime="2026-07-09 18:23:23.64131677 +0000 UTC m=+1225.728566074"
Jul 09 18:23:54.627251 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:54.627195    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:23:57.024446 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.024397    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:23:57.028218 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.028200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.031626 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.031598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-fw4bd\""
Jul 09 18:23:57.031748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.031652    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 18:23:57.038814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.038776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:23:57.087468 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.087432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.087646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.087480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.087646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.087533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjxz\" (UniqueName: \"kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.188763 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.188724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.188938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.188771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.188938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.188797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjxz\" (UniqueName: \"kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.189131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.189113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.191268 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.191217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.197023 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.196990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjxz\" (UniqueName: \"kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.339368 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.339270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.479392 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.479361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:23:57.481476 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:23:57.481449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc602d6_caee_40d8_91fb_1be182f37e46.slice/crio-0e66fb69fc86ce083dc69e5e08826d5d1911c1aa9869b0a5a9263cfd34dbd8aa WatchSource:0}: Error finding container 0e66fb69fc86ce083dc69e5e08826d5d1911c1aa9869b0a5a9263cfd34dbd8aa: Status 404 returned error can't find the container with id 0e66fb69fc86ce083dc69e5e08826d5d1911c1aa9869b0a5a9263cfd34dbd8aa
Jul 09 18:23:57.798086 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.798050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" event={"ID":"8fc602d6-caee-40d8-91fb-1be182f37e46","Type":"ContainerStarted","Data":"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"}
Jul 09 18:23:57.798301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.798099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" event={"ID":"8fc602d6-caee-40d8-91fb-1be182f37e46","Type":"ContainerStarted","Data":"0e66fb69fc86ce083dc69e5e08826d5d1911c1aa9869b0a5a9263cfd34dbd8aa"}
Jul 09 18:23:57.798301 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.798127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:23:57.817952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:23:57.817893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" podStartSLOduration=0.817877105 podStartE2EDuration="817.877105ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:23:57.815902018 +0000 UTC m=+1259.903151357" watchObservedRunningTime="2026-07-09 18:23:57.817877105 +0000 UTC m=+1259.905126409"
Jul 09 18:24:28.804472 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:28.804443    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:24:32.241788 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.241751    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:24:32.251021 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:32.250974    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:24:32.251021 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:32.251021 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:32.251021 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2" cmd=["/bin/sleep","15"]
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.251038    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" message="" err=<
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:32.251054    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerName="main" containerID="cri-o://05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"
Jul 09 18:24:32.251355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.251104    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerName="main" containerID="cri-o://05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2" gracePeriod=60
Jul 09 18:24:32.528739 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.528705    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:24:32.605115 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.605081    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrjxz\" (UniqueName: \"kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz\") pod \"8fc602d6-caee-40d8-91fb-1be182f37e46\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") "
Jul 09 18:24:32.605342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.605129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs\") pod \"8fc602d6-caee-40d8-91fb-1be182f37e46\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") "
Jul 09 18:24:32.605342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.605151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds\") pod \"8fc602d6-caee-40d8-91fb-1be182f37e46\" (UID: \"8fc602d6-caee-40d8-91fb-1be182f37e46\") "
Jul 09 18:24:32.605486 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.605439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8fc602d6-caee-40d8-91fb-1be182f37e46" (UID: "8fc602d6-caee-40d8-91fb-1be182f37e46"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:32.607300 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.607274    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc602d6-caee-40d8-91fb-1be182f37e46" (UID: "8fc602d6-caee-40d8-91fb-1be182f37e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:32.607409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.607330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz" (OuterVolumeSpecName: "kube-api-access-mrjxz") pod "8fc602d6-caee-40d8-91fb-1be182f37e46" (UID: "8fc602d6-caee-40d8-91fb-1be182f37e46"). InnerVolumeSpecName "kube-api-access-mrjxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:32.706633 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.706597    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc602d6-caee-40d8-91fb-1be182f37e46-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:32.706633 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.706626    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fc602d6-caee-40d8-91fb-1be182f37e46-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:32.706633 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.706636    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjxz\" (UniqueName: \"kubernetes.io/projected/8fc602d6-caee-40d8-91fb-1be182f37e46-kube-api-access-mrjxz\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:32.983493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.983457    2565 generic.go:356] "Generic (PLEG): container finished" podID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerID="05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2" exitCode=0
Jul 09 18:24:32.983697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.983501    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"
Jul 09 18:24:32.983697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.983549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" event={"ID":"8fc602d6-caee-40d8-91fb-1be182f37e46","Type":"ContainerDied","Data":"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"}
Jul 09 18:24:32.983697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.983598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2" event={"ID":"8fc602d6-caee-40d8-91fb-1be182f37e46","Type":"ContainerDied","Data":"0e66fb69fc86ce083dc69e5e08826d5d1911c1aa9869b0a5a9263cfd34dbd8aa"}
Jul 09 18:24:32.983697 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.983620    2565 scope.go:117] "RemoveContainer" containerID="05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"
Jul 09 18:24:32.993162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.993147    2565 scope.go:117] "RemoveContainer" containerID="05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"
Jul 09 18:24:32.993443 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:32.993423    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2\": container with ID starting with 05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2 not found: ID does not exist" containerID="05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"
Jul 09 18:24:32.993487 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:32.993454    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2"} err="failed to get container status \"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2\": rpc error: code = NotFound desc = could not find container \"05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2\": container with ID starting with 05e872ed64510a86a44b388515752cffeef5e882fb67c001a6d4d4549e956ac2 not found: ID does not exist"
Jul 09 18:24:33.006547 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:33.006516    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:24:33.010515 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:33.010455    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schelnfv2"]
Jul 09 18:24:34.481949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:34.481911    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" path="/var/lib/kubelet/pods/8fc602d6-caee-40d8-91fb-1be182f37e46/volumes"
Jul 09 18:24:52.398824 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.398785    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:24:52.408620 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:52.408575    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:24:52.408620 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:52.408620 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:52.408620 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9" cmd=["/bin/sleep","15"]
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.408643    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" message="" err=<
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:52.408659    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" podUID="b743270a-d647-4839-8938-35bde02d2463" containerName="main" containerID="cri-o://f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"
Jul 09 18:24:52.408916 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.408721    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" podUID="b743270a-d647-4839-8938-35bde02d2463" containerName="main" containerID="cri-o://f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9" gracePeriod=60
Jul 09 18:24:52.460531 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:52.460489    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb743270a_d647_4839_8938_35bde02d2463.slice/crio-conmon-f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb743270a_d647_4839_8938_35bde02d2463.slice/crio-f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9.scope\": RecentStats: unable to find data in memory cache]"
Jul 09 18:24:52.661027 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.660961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:24:52.780459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.780409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8vt9\" (UniqueName: \"kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9\") pod \"b743270a-d647-4839-8938-35bde02d2463\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") "
Jul 09 18:24:52.780665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.780486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds\") pod \"b743270a-d647-4839-8938-35bde02d2463\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") "
Jul 09 18:24:52.780665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.780508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs\") pod \"b743270a-d647-4839-8938-35bde02d2463\" (UID: \"b743270a-d647-4839-8938-35bde02d2463\") "
Jul 09 18:24:52.780784 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.780759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b743270a-d647-4839-8938-35bde02d2463" (UID: "b743270a-d647-4839-8938-35bde02d2463"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:52.782699 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.782670    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9" (OuterVolumeSpecName: "kube-api-access-t8vt9") pod "b743270a-d647-4839-8938-35bde02d2463" (UID: "b743270a-d647-4839-8938-35bde02d2463"). InnerVolumeSpecName "kube-api-access-t8vt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:52.782806 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.782698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b743270a-d647-4839-8938-35bde02d2463" (UID: "b743270a-d647-4839-8938-35bde02d2463"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:52.881759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.881716    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vt9\" (UniqueName: \"kubernetes.io/projected/b743270a-d647-4839-8938-35bde02d2463-kube-api-access-t8vt9\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:52.881759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.881754    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b743270a-d647-4839-8938-35bde02d2463-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:52.881759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:52.881765    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b743270a-d647-4839-8938-35bde02d2463-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:24:53.089422 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.089388    2565 generic.go:356] "Generic (PLEG): container finished" podID="b743270a-d647-4839-8938-35bde02d2463" containerID="f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9" exitCode=0
Jul 09 18:24:53.089606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.089433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" event={"ID":"b743270a-d647-4839-8938-35bde02d2463","Type":"ContainerDied","Data":"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"}
Jul 09 18:24:53.089606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.089438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"
Jul 09 18:24:53.089606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.089473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk" event={"ID":"b743270a-d647-4839-8938-35bde02d2463","Type":"ContainerDied","Data":"d90cc22c7eb57f0517e4cf1de61d89f31e3227349acb151afe33a29fb474110f"}
Jul 09 18:24:53.089606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.089493    2565 scope.go:117] "RemoveContainer" containerID="f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"
Jul 09 18:24:53.099119 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.099101    2565 scope.go:117] "RemoveContainer" containerID="f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"
Jul 09 18:24:53.099430 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:24:53.099411    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9\": container with ID starting with f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9 not found: ID does not exist" containerID="f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"
Jul 09 18:24:53.099469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.099439    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9"} err="failed to get container status \"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9\": rpc error: code = NotFound desc = could not find container \"f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9\": container with ID starting with f9fa7fde5800d12bf460678b319938be736290cdb1fe0faf61d270aa8ce33bb9 not found: ID does not exist"
Jul 09 18:24:53.112347 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.112323    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:24:53.117477 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:53.117457    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepcljk"]
Jul 09 18:24:54.482010 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:24:54.481979    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b743270a-d647-4839-8938-35bde02d2463" path="/var/lib/kubelet/pods/b743270a-d647-4839-8938-35bde02d2463/volumes"
Jul 09 18:25:24.128297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128206    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:25:24.128707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128663    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b743270a-d647-4839-8938-35bde02d2463" containerName="main"
Jul 09 18:25:24.128707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128677    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b743270a-d647-4839-8938-35bde02d2463" containerName="main"
Jul 09 18:25:24.128707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128691    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerName="main"
Jul 09 18:25:24.128707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128697    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerName="main"
Jul 09 18:25:24.128851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128772    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc602d6-caee-40d8-91fb-1be182f37e46" containerName="main"
Jul 09 18:25:24.128851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.128787    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b743270a-d647-4839-8938-35bde02d2463" containerName="main"
Jul 09 18:25:24.133405 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.133387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.136117 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.136095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:25:24.136319 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.136165    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:25:24.136861 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.136844    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:25:24.137017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.136993    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-776dq\""
Jul 09 18:25:24.137134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.137036    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:25:24.137134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.137086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:25:24.145294 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.145265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:25:24.230950 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.230908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mft5b\" (UniqueName: \"kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.231125 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.230959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.231125 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.231035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.231125 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.231109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332503 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mft5b\" (UniqueName: \"kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332503 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332773 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332773 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.332987 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.332965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.335221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.335195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.342974 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.342945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mft5b\" (UniqueName: \"kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.444706 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.444675    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:24.588857 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:24.588826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:25:24.590976 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:25:24.590952    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61ce832_2d6a_46c2_ae91_9fe988127a16.slice/crio-4a2d4cf9af7b8db6c71f2544120e424899625107e6cdb217a67e223183505648 WatchSource:0}: Error finding container 4a2d4cf9af7b8db6c71f2544120e424899625107e6cdb217a67e223183505648: Status 404 returned error can't find the container with id 4a2d4cf9af7b8db6c71f2544120e424899625107e6cdb217a67e223183505648
Jul 09 18:25:25.253703 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:25.253662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerStarted","Data":"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"}
Jul 09 18:25:25.254103 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:25.253726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerStarted","Data":"4a2d4cf9af7b8db6c71f2544120e424899625107e6cdb217a67e223183505648"}
Jul 09 18:25:26.261312 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:26.261274    2565 generic.go:356] "Generic (PLEG): container finished" podID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerID="55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d" exitCode=0
Jul 09 18:25:26.261724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:26.261362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerDied","Data":"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"}
Jul 09 18:25:27.269111 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:27.269079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerStarted","Data":"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"}
Jul 09 18:25:27.269549 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:27.269221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:25:27.299985 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:27.299933    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" podStartSLOduration=2.638567076 podStartE2EDuration="3.299918252s" totalImagesPullingTime="0s" totalInitContainerRuntime="661.351176ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:27.298976714 +0000 UTC m=+1349.386226019" watchObservedRunningTime="2026-07-09 18:25:27.299918252 +0000 UTC m=+1349.387167589"
Jul 09 18:25:58.276074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:25:58.276041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:26:43.953411 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:43.953363    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:43.958284 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:43.958165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:43.964423 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:43.964386    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:26:43.964574 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:43.964534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-mg5cs\""
Jul 09 18:26:43.976704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:43.976664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:44.042751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042751 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.042999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.042997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxl8\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143505 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143505 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxl8\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.143886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.143904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144162 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.144126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144286 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.144263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144357 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.144296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.144527 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.144509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.146039 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.146019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.146283 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.146264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.154439 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.154412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.154705 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.154682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxl8\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8\") pod \"router-gateway-1-openshift-default-6679d9568-wpxqk\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.272165 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.272058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:44.442883 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.442681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:44.445734 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:26:44.445672    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1409d2c2_41be_40eb_83dc_e23c30890b22.slice/crio-790e2b85737d5ced7117e2f6624972aa9f63943498cd5b065c931b3ea2f9e8ec WatchSource:0}: Error finding container 790e2b85737d5ced7117e2f6624972aa9f63943498cd5b065c931b3ea2f9e8ec: Status 404 returned error can't find the container with id 790e2b85737d5ced7117e2f6624972aa9f63943498cd5b065c931b3ea2f9e8ec
Jul 09 18:26:44.448326 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.448277    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:26:44.448435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.448364    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:26:44.448435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.448395    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:26:44.711707 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.711663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" event={"ID":"1409d2c2-41be-40eb-83dc-e23c30890b22","Type":"ContainerStarted","Data":"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"}
Jul 09 18:26:44.711958 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.711733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" event={"ID":"1409d2c2-41be-40eb-83dc-e23c30890b22","Type":"ContainerStarted","Data":"790e2b85737d5ced7117e2f6624972aa9f63943498cd5b065c931b3ea2f9e8ec"}
Jul 09 18:26:44.736673 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.736589    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" podStartSLOduration=1.736563396 podStartE2EDuration="1.736563396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:44.734974326 +0000 UTC m=+1426.822223632" watchObservedRunningTime="2026-07-09 18:26:44.736563396 +0000 UTC m=+1426.823812697"
Jul 09 18:26:44.775452 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:44.775334    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:45.272646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:45.272607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:45.278671 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:45.278641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:45.718494 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:45.718455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:45.720343 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:45.720312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:46.723483 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:46.723444    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" podUID="1409d2c2-41be-40eb-83dc-e23c30890b22" containerName="istio-proxy" containerID="cri-o://aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897" gracePeriod=30
Jul 09 18:26:51.992120 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:51.992050    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:52.123097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123342 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123335    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fxl8\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:52.123685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert\") pod \"1409d2c2-41be-40eb-83dc-e23c30890b22\" (UID: \"1409d2c2-41be-40eb-83dc-e23c30890b22\") "
Jul 09 18:26:52.123685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:52.123896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123710    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.123896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123728    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.124073 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.123881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:52.124155 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.124133    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data" (OuterVolumeSpecName: "istio-data") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:52.124321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.124287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:52.125647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.125615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:52.125922 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.125904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:26:52.126082 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.126062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token" (OuterVolumeSpecName: "istio-token") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:52.126143 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.126075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8" (OuterVolumeSpecName: "kube-api-access-4fxl8") pod "1409d2c2-41be-40eb-83dc-e23c30890b22" (UID: "1409d2c2-41be-40eb-83dc-e23c30890b22"). InnerVolumeSpecName "kube-api-access-4fxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:52.224938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224905    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.224938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224936    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.224938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224945    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.225211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224953    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.225211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224962    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxl8\" (UniqueName: \"kubernetes.io/projected/1409d2c2-41be-40eb-83dc-e23c30890b22-kube-api-access-4fxl8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.225211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224971    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409d2c2-41be-40eb-83dc-e23c30890b22-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.225211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.224980    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409d2c2-41be-40eb-83dc-e23c30890b22-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:26:52.763909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.763807    2565 generic.go:356] "Generic (PLEG): container finished" podID="1409d2c2-41be-40eb-83dc-e23c30890b22" containerID="aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897" exitCode=0
Jul 09 18:26:52.763909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.763859    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"
Jul 09 18:26:52.763909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.763892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" event={"ID":"1409d2c2-41be-40eb-83dc-e23c30890b22","Type":"ContainerDied","Data":"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"}
Jul 09 18:26:52.764204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.763947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk" event={"ID":"1409d2c2-41be-40eb-83dc-e23c30890b22","Type":"ContainerDied","Data":"790e2b85737d5ced7117e2f6624972aa9f63943498cd5b065c931b3ea2f9e8ec"}
Jul 09 18:26:52.764204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.763972    2565 scope.go:117] "RemoveContainer" containerID="aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"
Jul 09 18:26:52.772901 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.772879    2565 scope.go:117] "RemoveContainer" containerID="aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"
Jul 09 18:26:52.773195 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:26:52.773171    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897\": container with ID starting with aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897 not found: ID does not exist" containerID="aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"
Jul 09 18:26:52.773295 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.773202    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897"} err="failed to get container status \"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897\": rpc error: code = NotFound desc = could not find container \"aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897\": container with ID starting with aa88ea193eb2ca4969f43df7e1740fb7540dd207bf870ad347ca2768799a0897 not found: ID does not exist"
Jul 09 18:26:52.784046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.784021    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:52.790400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:52.790378    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-wpxqk"]
Jul 09 18:26:54.483491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:26:54.483447    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1409d2c2-41be-40eb-83dc-e23c30890b22" path="/var/lib/kubelet/pods/1409d2c2-41be-40eb-83dc-e23c30890b22/volumes"
Jul 09 18:27:26.991684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:26.991647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:27:27.003060 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:27:27.003021    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:27:27.003060 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:27:27.003060 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:27:27.003060 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a" cmd=["/bin/sleep","15"]
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.003077    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" message="" err=<
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:27:27.003094    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="main" containerID="cri-o://bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"
Jul 09 18:27:27.003395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.003140    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="main" containerID="cri-o://bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a" gracePeriod=60
Jul 09 18:27:27.249489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.249432    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:27:27.357850 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.357815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs\") pod \"f61ce832-2d6a-46c2-ae91-9fe988127a16\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") "
Jul 09 18:27:27.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.357869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds\") pod \"f61ce832-2d6a-46c2-ae91-9fe988127a16\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") "
Jul 09 18:27:27.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.357922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mft5b\" (UniqueName: \"kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b\") pod \"f61ce832-2d6a-46c2-ae91-9fe988127a16\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") "
Jul 09 18:27:27.358019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.357980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location\") pod \"f61ce832-2d6a-46c2-ae91-9fe988127a16\" (UID: \"f61ce832-2d6a-46c2-ae91-9fe988127a16\") "
Jul 09 18:27:27.358182 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.358154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f61ce832-2d6a-46c2-ae91-9fe988127a16" (UID: "f61ce832-2d6a-46c2-ae91-9fe988127a16"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:27.358742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.358720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f61ce832-2d6a-46c2-ae91-9fe988127a16" (UID: "f61ce832-2d6a-46c2-ae91-9fe988127a16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:27.360134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.360115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b" (OuterVolumeSpecName: "kube-api-access-mft5b") pod "f61ce832-2d6a-46c2-ae91-9fe988127a16" (UID: "f61ce832-2d6a-46c2-ae91-9fe988127a16"). InnerVolumeSpecName "kube-api-access-mft5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:27:27.360200 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.360139    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f61ce832-2d6a-46c2-ae91-9fe988127a16" (UID: "f61ce832-2d6a-46c2-ae91-9fe988127a16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:27.458630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.458595    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mft5b\" (UniqueName: \"kubernetes.io/projected/f61ce832-2d6a-46c2-ae91-9fe988127a16-kube-api-access-mft5b\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:27:27.458630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.458624    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:27:27.458630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.458634    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61ce832-2d6a-46c2-ae91-9fe988127a16-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:27:27.458876 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.458643    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f61ce832-2d6a-46c2-ae91-9fe988127a16-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:27:27.948728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.948695    2565 generic.go:356] "Generic (PLEG): container finished" podID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerID="bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a" exitCode=0
Jul 09 18:27:27.948945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.948755    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"
Jul 09 18:27:27.948945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.948778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerDied","Data":"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"}
Jul 09 18:27:27.948945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.948840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b" event={"ID":"f61ce832-2d6a-46c2-ae91-9fe988127a16","Type":"ContainerDied","Data":"4a2d4cf9af7b8db6c71f2544120e424899625107e6cdb217a67e223183505648"}
Jul 09 18:27:27.948945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.948864    2565 scope.go:117] "RemoveContainer" containerID="bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"
Jul 09 18:27:27.958961 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.958937    2565 scope.go:117] "RemoveContainer" containerID="55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"
Jul 09 18:27:27.967648 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.967632    2565 scope.go:117] "RemoveContainer" containerID="bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"
Jul 09 18:27:27.967928 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:27:27.967906    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a\": container with ID starting with bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a not found: ID does not exist" containerID="bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"
Jul 09 18:27:27.967983 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.967937    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a"} err="failed to get container status \"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a\": rpc error: code = NotFound desc = could not find container \"bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a\": container with ID starting with bf1aac2ca8576176bca4275fef9b0d59d9c090d59044939a3f01301fc427742a not found: ID does not exist"
Jul 09 18:27:27.967983 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.967953    2565 scope.go:117] "RemoveContainer" containerID="55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"
Jul 09 18:27:27.968206 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:27:27.968189    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d\": container with ID starting with 55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d not found: ID does not exist" containerID="55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"
Jul 09 18:27:27.968275 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.968212    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d"} err="failed to get container status \"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d\": rpc error: code = NotFound desc = could not find container \"55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d\": container with ID starting with 55b50ad1bcedcd987a449a984a9455d8dde5f4312b37bbc933471fcd77600a0d not found: ID does not exist"
Jul 09 18:27:27.973288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.973261    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:27:27.977963 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:27.977943    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelcs4b"]
Jul 09 18:27:28.483167 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:27:28.483136    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" path="/var/lib/kubelet/pods/f61ce832-2d6a-46c2-ae91-9fe988127a16/volumes"
Jul 09 18:28:02.370748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.370703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371124    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="main"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="main"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="storage-initializer"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371160    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="storage-initializer"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371177    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1409d2c2-41be-40eb-83dc-e23c30890b22" containerName="istio-proxy"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371185    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1409d2c2-41be-40eb-83dc-e23c30890b22" containerName="istio-proxy"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371261    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f61ce832-2d6a-46c2-ae91-9fe988127a16" containerName="main"
Jul 09 18:28:02.371269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.371273    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1409d2c2-41be-40eb-83dc-e23c30890b22" containerName="istio-proxy"
Jul 09 18:28:02.374520 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.374499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.378780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.378740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-r2n8m\""
Jul 09 18:28:02.378780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.378746    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:28:02.378780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.378752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:28:02.379092 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.379077    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:28:02.379586 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.379565    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:28:02.384117 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.384094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:28:02.388479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.388447    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:28:02.488873 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.488822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.489077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.488894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.489077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.488932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.489077 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.488956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rth2\" (UniqueName: \"kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590371 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rth2\" (UniqueName: \"kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590704 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.590824 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.590789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.592878 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.592857    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.598989 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.598964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rth2\" (UniqueName: \"kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.685560 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.685450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:02.841211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:02.841191    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:28:02.843638 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:28:02.843602    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bc57f8_d1f8_43d6_9586_394f1b84b3a5.slice/crio-e19d247c9e76109054430720c6b60fb5230ee09aad3dca7b15962091ca830d75 WatchSource:0}: Error finding container e19d247c9e76109054430720c6b60fb5230ee09aad3dca7b15962091ca830d75: Status 404 returned error can't find the container with id e19d247c9e76109054430720c6b60fb5230ee09aad3dca7b15962091ca830d75
Jul 09 18:28:03.145802 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:03.145754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerStarted","Data":"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"}
Jul 09 18:28:03.146039 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:03.145815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerStarted","Data":"e19d247c9e76109054430720c6b60fb5230ee09aad3dca7b15962091ca830d75"}
Jul 09 18:28:04.152844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:04.152805    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerID="193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd" exitCode=0
Jul 09 18:28:04.153388 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:04.152894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerDied","Data":"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"}
Jul 09 18:28:05.160219 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:05.160182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerStarted","Data":"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"}
Jul 09 18:28:05.160655 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:05.160312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:28:05.185909 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:05.185853    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" podStartSLOduration=2.452409109 podStartE2EDuration="3.185837301s" totalImagesPullingTime="0s" totalInitContainerRuntime="733.428192ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:05.182777893 +0000 UTC m=+1507.270027198" watchObservedRunningTime="2026-07-09 18:28:05.185837301 +0000 UTC m=+1507.273086605"
Jul 09 18:28:09.756640 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.756609    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:28:09.758564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.758542    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:28:09.761225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.761206    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:28:09.763409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.763388    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:28:09.768198 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.768181    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:28:09.769752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:09.769735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:28:36.168783 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:28:36.168754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:29:33.757612 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:33.757571    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:29:33.766789 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:29:33.766753    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:29:33.766789 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:29:33.766789 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:29:33.766789 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f" cmd=["/bin/sleep","15"]
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:33.766814    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" message="" err=<
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:29:33.766829    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="main" containerID="cri-o://c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"
Jul 09 18:29:33.767078 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:33.766878    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="main" containerID="cri-o://c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f" gracePeriod=60
Jul 09 18:29:34.023630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.023604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:29:34.117020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.116981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds\") pod \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") "
Jul 09 18:29:34.117196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.117045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rth2\" (UniqueName: \"kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2\") pod \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") "
Jul 09 18:29:34.117196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.117091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location\") pod \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") "
Jul 09 18:29:34.117196 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.117139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs\") pod \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\" (UID: \"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5\") "
Jul 09 18:29:34.117395 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.117335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" (UID: "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:34.117886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.117860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" (UID: "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:34.119262 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.119224    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" (UID: "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:29:34.119387 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.119362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2" (OuterVolumeSpecName: "kube-api-access-9rth2") pod "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" (UID: "c4bc57f8-d1f8-43d6-9586-394f1b84b3a5"). InnerVolumeSpecName "kube-api-access-9rth2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:29:34.218683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.218634    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:29:34.218683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.218668    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:29:34.218683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.218677    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rth2\" (UniqueName: \"kubernetes.io/projected/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kube-api-access-9rth2\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:29:34.218683 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.218686    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:29:34.668170 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.668133    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerID="c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f" exitCode=0
Jul 09 18:29:34.668390 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.668184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"
Jul 09 18:29:34.668390 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.668218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerDied","Data":"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"}
Jul 09 18:29:34.668390 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.668296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms" event={"ID":"c4bc57f8-d1f8-43d6-9586-394f1b84b3a5","Type":"ContainerDied","Data":"e19d247c9e76109054430720c6b60fb5230ee09aad3dca7b15962091ca830d75"}
Jul 09 18:29:34.668390 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.668318    2565 scope.go:117] "RemoveContainer" containerID="c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"
Jul 09 18:29:34.678414 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.678292    2565 scope.go:117] "RemoveContainer" containerID="193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"
Jul 09 18:29:34.687999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.687972    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:29:34.688205 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.688191    2565 scope.go:117] "RemoveContainer" containerID="c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"
Jul 09 18:29:34.688493 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:29:34.688471    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f\": container with ID starting with c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f not found: ID does not exist" containerID="c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"
Jul 09 18:29:34.688599 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.688499    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f"} err="failed to get container status \"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f\": rpc error: code = NotFound desc = could not find container \"c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f\": container with ID starting with c61d842a91c49a57e5f0e7a712983a83e5ddb6f8f3b81b7d401475e6b782220f not found: ID does not exist"
Jul 09 18:29:34.688599 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.688516    2565 scope.go:117] "RemoveContainer" containerID="193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"
Jul 09 18:29:34.688740 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:29:34.688718    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd\": container with ID starting with 193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd not found: ID does not exist" containerID="193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"
Jul 09 18:29:34.688793 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.688753    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd"} err="failed to get container status \"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd\": rpc error: code = NotFound desc = could not find container \"193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd\": container with ID starting with 193dd84b1bddb264b662c736f77c9fcb8c4f6562cc7ae62c2f99c46f5a34facd not found: ID does not exist"
Jul 09 18:29:34.692939 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:34.692917    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6hjms"]
Jul 09 18:29:36.481928 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:36.481893    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" path="/var/lib/kubelet/pods/c4bc57f8-d1f8-43d6-9586-394f1b84b3a5/volumes"
Jul 09 18:29:56.971943 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.971908    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"]
Jul 09 18:29:56.972426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.972299    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="main"
Jul 09 18:29:56.972426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.972311    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="main"
Jul 09 18:29:56.972426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.972323    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="storage-initializer"
Jul 09 18:29:56.972426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.972329    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="storage-initializer"
Jul 09 18:29:56.972426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.972395    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4bc57f8-d1f8-43d6-9586-394f1b84b3a5" containerName="main"
Jul 09 18:29:56.976922 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.976905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:29:56.979768 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.979740    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:29:56.979768 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.979746    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:29:56.979948 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.979791    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:29:56.983795 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:56.983764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"]
Jul 09 18:29:57.120887 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:57.120845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rklfj\" (UniqueName: \"kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:29:57.222253 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:57.222153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rklfj\" (UniqueName: \"kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:29:57.231662 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:29:57.231637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rklfj\" (UniqueName: \"kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:01.759317 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:01.759278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:01.767550 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:01.767512    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:01.860125 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:01.860088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:01.860321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:01.860198    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:01.864427 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:01.864406    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") DevicePath \"csi-f4bb4a44099ee514702a90cda592b016a8b02fc81290100fb169852ffee5f55e\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:02.091694 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.091666    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d55ac04b2c83230ef464e6ac64fdc05efc871de6bd51fec133c9f911cb36e9ef/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:02.107466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.107436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"e2e-pvc-model-download-24fsq\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:02.404744 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.404657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:30:02.412034 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.412014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:02.545862 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.545839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"]
Jul 09 18:30:02.548452 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:30:02.548424    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45d95cc2_d0b6_40a5_88e2_e4dcf0f00eb5.slice/crio-2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a WatchSource:0}: Error finding container 2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a: Status 404 returned error can't find the container with id 2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a
Jul 09 18:30:02.817022 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.816988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq" event={"ID":"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5","Type":"ContainerStarted","Data":"2fa2f747096b55b3d93faa4f990071d6bfe767e1b9ccc3bd85e140288985e234"}
Jul 09 18:30:02.817491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.817031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq" event={"ID":"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5","Type":"ContainerStarted","Data":"2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a"}
Jul 09 18:30:02.837144 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:02.837093    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq" podStartSLOduration=9.837075879 podStartE2EDuration="9.837075879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:02.833388841 +0000 UTC m=+1624.920638142" watchObservedRunningTime="2026-07-09 18:30:02.837075879 +0000 UTC m=+1624.924325182"
Jul 09 18:30:08.862616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:08.862581    2565 generic.go:356] "Generic (PLEG): container finished" podID="45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" containerID="2fa2f747096b55b3d93faa4f990071d6bfe767e1b9ccc3bd85e140288985e234" exitCode=0
Jul 09 18:30:08.863043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:08.862655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq" event={"ID":"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5","Type":"ContainerDied","Data":"2fa2f747096b55b3d93faa4f990071d6bfe767e1b9ccc3bd85e140288985e234"}
Jul 09 18:30:10.023797 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.023772    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:10.033875 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.033855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") "
Jul 09 18:30:10.033991 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.033894    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rklfj\" (UniqueName: \"kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj\") pod \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\" (UID: \"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5\") "
Jul 09 18:30:10.036017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.035985    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj" (OuterVolumeSpecName: "kube-api-access-rklfj") pod "45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" (UID: "45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5"). InnerVolumeSpecName "kube-api-access-rklfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:10.037409 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.037389    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "model-storage") pod "45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" (UID: "45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:30:10.134684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.134599    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-141-82.ec2.internal\" "
Jul 09 18:30:10.134684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.134631    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rklfj\" (UniqueName: \"kubernetes.io/projected/45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5-kube-api-access-rklfj\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:30:10.240074 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.240054    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529") on node "ip-10-0-141-82.ec2.internal"
Jul 09 18:30:10.336360 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.336326    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"csi-f4bb4a44099ee514702a90cda592b016a8b02fc81290100fb169852ffee5f55e\""
Jul 09 18:30:10.874057 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.873971    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq"
Jul 09 18:30:10.874211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.873971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-24fsq" event={"ID":"45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5","Type":"ContainerDied","Data":"2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a"}
Jul 09 18:30:10.874211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:10.874091    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2746c0a485821f2910ab6f53cf6f17123b9fa19675083364c1506fd923e15b2a"
Jul 09 18:30:25.363792 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.363760    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:30:25.364288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.364175    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" containerName="storage-initializer"
Jul 09 18:30:25.364288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.364187    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" containerName="storage-initializer"
Jul 09 18:30:25.364288 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.364272    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5" containerName="storage-initializer"
Jul 09 18:30:25.367516 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.367497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.372658 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.372626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:30:25.372658 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.372642    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:30:25.372971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.372946    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:30:25.373105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.372955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:30:25.373330 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.373312    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-grwzf\""
Jul 09 18:30:25.382249 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.382208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:30:25.464542 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.464505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.464753 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.464574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.464753 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.464654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkg6\" (UniqueName: \"kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.565929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.565880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkg6\" (UniqueName: \"kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.566143 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.566062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.566215 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.566146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.566453 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.566436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.568590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.568562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.576024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.576001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkg6\" (UniqueName: \"kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.676799 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.676708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.817204 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.817180    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:30:25.819107 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:30:25.819078    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed4bc16_6fad_4116_b735_691a2abb40a6.slice/crio-ff2eb6c43dbefc20753d782a6d778a4e789d9a892d16e1c5d2ecd0d4c9b4ce4d WatchSource:0}: Error finding container ff2eb6c43dbefc20753d782a6d778a4e789d9a892d16e1c5d2ecd0d4c9b4ce4d: Status 404 returned error can't find the container with id ff2eb6c43dbefc20753d782a6d778a4e789d9a892d16e1c5d2ecd0d4c9b4ce4d
Jul 09 18:30:25.960735 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.960693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" event={"ID":"eed4bc16-6fad-4116-b735-691a2abb40a6","Type":"ContainerStarted","Data":"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"}
Jul 09 18:30:25.960937 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.960762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:30:25.960937 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.960785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" event={"ID":"eed4bc16-6fad-4116-b735-691a2abb40a6","Type":"ContainerStarted","Data":"ff2eb6c43dbefc20753d782a6d778a4e789d9a892d16e1c5d2ecd0d4c9b4ce4d"}
Jul 09 18:30:25.979917 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:25.979849    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" podStartSLOduration=0.979833223 podStartE2EDuration="979.833223ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:25.978917919 +0000 UTC m=+1648.066167223" watchObservedRunningTime="2026-07-09 18:30:25.979833223 +0000 UTC m=+1648.067082528"
Jul 09 18:30:56.968206 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:30:56.968175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:32:35.036475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.036441    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:32:35.046021 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:32:35.045987    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:32:35.046021 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:32:35.046021 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:32:35.046021 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747" cmd=["/bin/sleep","15"]
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.046034    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" message="" err=<
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:32:35.046046    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerName="main" containerID="cri-o://ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"
Jul 09 18:32:35.046397 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.046100    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerName="main" containerID="cri-o://ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747" gracePeriod=60
Jul 09 18:32:35.301535 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.301509    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:32:35.320172 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.320137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs\") pod \"eed4bc16-6fad-4116-b735-691a2abb40a6\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") "
Jul 09 18:32:35.320369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.320290    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds\") pod \"eed4bc16-6fad-4116-b735-691a2abb40a6\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") "
Jul 09 18:32:35.320369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.320326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khkg6\" (UniqueName: \"kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6\") pod \"eed4bc16-6fad-4116-b735-691a2abb40a6\" (UID: \"eed4bc16-6fad-4116-b735-691a2abb40a6\") "
Jul 09 18:32:35.320623 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.320568    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "eed4bc16-6fad-4116-b735-691a2abb40a6" (UID: "eed4bc16-6fad-4116-b735-691a2abb40a6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:35.320758 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.320688    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eed4bc16-6fad-4116-b735-691a2abb40a6-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:32:35.322802 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.322755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eed4bc16-6fad-4116-b735-691a2abb40a6" (UID: "eed4bc16-6fad-4116-b735-691a2abb40a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:32:35.323684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.323659    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6" (OuterVolumeSpecName: "kube-api-access-khkg6") pod "eed4bc16-6fad-4116-b735-691a2abb40a6" (UID: "eed4bc16-6fad-4116-b735-691a2abb40a6"). InnerVolumeSpecName "kube-api-access-khkg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:32:35.421373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.421337    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed4bc16-6fad-4116-b735-691a2abb40a6-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:32:35.421373 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.421366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkg6\" (UniqueName: \"kubernetes.io/projected/eed4bc16-6fad-4116-b735-691a2abb40a6-kube-api-access-khkg6\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:32:35.643469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.643374    2565 generic.go:356] "Generic (PLEG): container finished" podID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerID="ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747" exitCode=0
Jul 09 18:32:35.643469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.643421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"
Jul 09 18:32:35.643469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.643428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" event={"ID":"eed4bc16-6fad-4116-b735-691a2abb40a6","Type":"ContainerDied","Data":"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"}
Jul 09 18:32:35.643469 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.643470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728" event={"ID":"eed4bc16-6fad-4116-b735-691a2abb40a6","Type":"ContainerDied","Data":"ff2eb6c43dbefc20753d782a6d778a4e789d9a892d16e1c5d2ecd0d4c9b4ce4d"}
Jul 09 18:32:35.643765 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.643490    2565 scope.go:117] "RemoveContainer" containerID="ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"
Jul 09 18:32:35.654269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.654211    2565 scope.go:117] "RemoveContainer" containerID="ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"
Jul 09 18:32:35.654567 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:32:35.654551    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747\": container with ID starting with ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747 not found: ID does not exist" containerID="ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"
Jul 09 18:32:35.654616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.654576    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747"} err="failed to get container status \"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747\": rpc error: code = NotFound desc = could not find container \"ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747\": container with ID starting with ce8ef204ddf51e97c4e74bfd542c11bc6ad6b803648455836483594dff8ae747 not found: ID does not exist"
Jul 09 18:32:35.665885 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.665861    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:32:35.672158 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:35.672130    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekn728"]
Jul 09 18:32:36.482071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:32:36.482027    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" path="/var/lib/kubelet/pods/eed4bc16-6fad-4116-b735-691a2abb40a6/volumes"
Jul 09 18:33:00.425769 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.425730    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:33:00.426222 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.426112    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerName="main"
Jul 09 18:33:00.426222 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.426123    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerName="main"
Jul 09 18:33:00.426222 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.426179    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed4bc16-6fad-4116-b735-691a2abb40a6" containerName="main"
Jul 09 18:33:00.429157 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.429138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.434494 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.434470    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:33:00.435709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.435540    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ft2xm\""
Jul 09 18:33:00.435709 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.435549    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 18:33:00.435859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.435562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:33:00.435859 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.435686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:33:00.444058 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.444032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:33:00.525806 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.525761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.525996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.525820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.525996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.525885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvsn\" (UniqueName: \"kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.626893 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.626852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.627072 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.626905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.627072 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.626946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvsn\" (UniqueName: \"kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.627419 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.627392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.629404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.629383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.636191 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.636163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvsn\" (UniqueName: \"kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.738217 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.738173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:00.879040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:00.878992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:33:00.881331 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:33:00.881290    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823e08fd_1d18_45dc_89ca_7a4212f46c38.slice/crio-2aa24f0a4e6dc71fcc96c063ebe4c13bd444f3161f750929ca048813c03f47b3 WatchSource:0}: Error finding container 2aa24f0a4e6dc71fcc96c063ebe4c13bd444f3161f750929ca048813c03f47b3: Status 404 returned error can't find the container with id 2aa24f0a4e6dc71fcc96c063ebe4c13bd444f3161f750929ca048813c03f47b3
Jul 09 18:33:01.779901 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:01.779862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" event={"ID":"823e08fd-1d18-45dc-89ca-7a4212f46c38","Type":"ContainerStarted","Data":"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"}
Jul 09 18:33:01.780431 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:01.779948    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:33:01.780431 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:01.779965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" event={"ID":"823e08fd-1d18-45dc-89ca-7a4212f46c38","Type":"ContainerStarted","Data":"2aa24f0a4e6dc71fcc96c063ebe4c13bd444f3161f750929ca048813c03f47b3"}
Jul 09 18:33:01.801426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:01.801373    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" podStartSLOduration=1.8013587279999999 podStartE2EDuration="1.801358728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:01.799193011 +0000 UTC m=+1803.886442316" watchObservedRunningTime="2026-07-09 18:33:01.801358728 +0000 UTC m=+1803.888608032"
Jul 09 18:33:09.792316 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.792207    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:33:09.795530 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.795501    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:33:09.796628 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.796608    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:33:09.800110 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.800086    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:33:09.807477 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.807382    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:33:09.811536 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:09.811514    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:33:32.786067 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:33:32.786035    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:34:01.736349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.736310    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:01.739763 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.739742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.743631 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.743609    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-mg5cs\""
Jul 09 18:34:01.743817 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.743796    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:34:01.765919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.765886    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:01.867054 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867277 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867277 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867277 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867277 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867277 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.867497 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.867393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8d7\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968380 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968380 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8d7\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968651 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968993 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968993 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.968993 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.968961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.969142 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.969112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.969315 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.969293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.970905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.970882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.971134 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.971115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.977407 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.977384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:01.977734 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:01.977716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8d7\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7\") pod \"router-gateway-1-openshift-default-6679d9568-b62dj\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:02.049913 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:02.049822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:02.196593 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:02.196564    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:02.198780 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:34:02.198742    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c2464d_4ba6_42cd_9ba0_e21613124b48.slice/crio-29e088ae1ac8afd2d14c5ac6ce5212987c47b64d6f6cf38e586ebe5b07a2e773 WatchSource:0}: Error finding container 29e088ae1ac8afd2d14c5ac6ce5212987c47b64d6f6cf38e586ebe5b07a2e773: Status 404 returned error can't find the container with id 29e088ae1ac8afd2d14c5ac6ce5212987c47b64d6f6cf38e586ebe5b07a2e773
Jul 09 18:34:02.201665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:02.201597    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:34:02.201780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:02.201681    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:34:02.201780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:02.201721    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:34:03.108912 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:03.108874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" event={"ID":"18c2464d-4ba6-42cd-9ba0-e21613124b48","Type":"ContainerStarted","Data":"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"}
Jul 09 18:34:03.109329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:03.108929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" event={"ID":"18c2464d-4ba6-42cd-9ba0-e21613124b48","Type":"ContainerStarted","Data":"29e088ae1ac8afd2d14c5ac6ce5212987c47b64d6f6cf38e586ebe5b07a2e773"}
Jul 09 18:34:03.132369 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:03.132306    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" podStartSLOduration=2.132290899 podStartE2EDuration="2.132290899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:03.131133419 +0000 UTC m=+1865.218382735" watchObservedRunningTime="2026-07-09 18:34:03.132290899 +0000 UTC m=+1865.219540204"
Jul 09 18:34:03.878942 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:03.878909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:04.051009 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:04.050967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:04.057400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:04.057375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:04.114499 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:04.114459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:04.115388 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:04.115364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:05.119379 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:05.119335    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" podUID="18c2464d-4ba6-42cd-9ba0-e21613124b48" containerName="istio-proxy" containerID="cri-o://9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad" gracePeriod=30
Jul 09 18:34:10.370733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.370707    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:10.547286 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547300    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj8d7\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547871 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547495    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547871 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:10.547871 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data\") pod \"18c2464d-4ba6-42cd-9ba0-e21613124b48\" (UID: \"18c2464d-4ba6-42cd-9ba0-e21613124b48\") "
Jul 09 18:34:10.547871 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:10.547871 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547741    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:10.548118 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data" (OuterVolumeSpecName: "istio-data") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:10.548118 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547900    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:34:10.548118 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547919    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.548118 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547940    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.548118 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.547955    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.549678 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.549650    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:10.549790 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.549687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token" (OuterVolumeSpecName: "istio-token") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:10.549790 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.549716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7" (OuterVolumeSpecName: "kube-api-access-lj8d7") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "kube-api-access-lj8d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:10.549790 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.549725    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18c2464d-4ba6-42cd-9ba0-e21613124b48" (UID: "18c2464d-4ba6-42cd-9ba0-e21613124b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:34:10.649349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649317    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.649349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649343    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c2464d-4ba6-42cd-9ba0-e21613124b48-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.649349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649352    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.649349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649360    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.649619 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649369    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c2464d-4ba6-42cd-9ba0-e21613124b48-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:10.649619 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:10.649377    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8d7\" (UniqueName: \"kubernetes.io/projected/18c2464d-4ba6-42cd-9ba0-e21613124b48-kube-api-access-lj8d7\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:34:11.161514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.161482    2565 generic.go:356] "Generic (PLEG): container finished" podID="18c2464d-4ba6-42cd-9ba0-e21613124b48" containerID="9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad" exitCode=0
Jul 09 18:34:11.161685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.161523    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"
Jul 09 18:34:11.161685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.161562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" event={"ID":"18c2464d-4ba6-42cd-9ba0-e21613124b48","Type":"ContainerDied","Data":"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"}
Jul 09 18:34:11.161685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.161628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj" event={"ID":"18c2464d-4ba6-42cd-9ba0-e21613124b48","Type":"ContainerDied","Data":"29e088ae1ac8afd2d14c5ac6ce5212987c47b64d6f6cf38e586ebe5b07a2e773"}
Jul 09 18:34:11.161685 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.161651    2565 scope.go:117] "RemoveContainer" containerID="9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"
Jul 09 18:34:11.173057 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.173031    2565 scope.go:117] "RemoveContainer" containerID="9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"
Jul 09 18:34:11.173358 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:34:11.173340    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad\": container with ID starting with 9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad not found: ID does not exist" containerID="9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"
Jul 09 18:34:11.173435 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.173367    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad"} err="failed to get container status \"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad\": rpc error: code = NotFound desc = could not find container \"9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad\": container with ID starting with 9bdd0592c58fa526a6160d8b2c59421f1d86c2996a1a73f94a1f535595acb0ad not found: ID does not exist"
Jul 09 18:34:11.187123 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.187095    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:11.191059 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:11.191034    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-b62dj"]
Jul 09 18:34:12.481935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:34:12.481901    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c2464d-4ba6-42cd-9ba0-e21613124b48" path="/var/lib/kubelet/pods/18c2464d-4ba6-42cd-9ba0-e21613124b48/volumes"
Jul 09 18:35:59.622929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:35:59.622880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:35:59.635108 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:35:59.635065    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:35:59.635108 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:59.635108 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:59.635108 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707" cmd=["/bin/sleep","15"]
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:35:59.635156    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" message="" err=<
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:35:59.635174    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerName="main" containerID="cri-o://e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"
Jul 09 18:35:59.635504 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:35:59.635224    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerName="main" containerID="cri-o://e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707" gracePeriod=60
Jul 09 18:35:59.912208 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:35:59.912166    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:36:00.098185 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.098148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfvsn\" (UniqueName: \"kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn\") pod \"823e08fd-1d18-45dc-89ca-7a4212f46c38\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") "
Jul 09 18:36:00.098411 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.098214    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs\") pod \"823e08fd-1d18-45dc-89ca-7a4212f46c38\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") "
Jul 09 18:36:00.098411 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.098264    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds\") pod \"823e08fd-1d18-45dc-89ca-7a4212f46c38\" (UID: \"823e08fd-1d18-45dc-89ca-7a4212f46c38\") "
Jul 09 18:36:00.098562 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.098540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "823e08fd-1d18-45dc-89ca-7a4212f46c38" (UID: "823e08fd-1d18-45dc-89ca-7a4212f46c38"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:00.100447 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.100420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "823e08fd-1d18-45dc-89ca-7a4212f46c38" (UID: "823e08fd-1d18-45dc-89ca-7a4212f46c38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:00.100447 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.100434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn" (OuterVolumeSpecName: "kube-api-access-gfvsn") pod "823e08fd-1d18-45dc-89ca-7a4212f46c38" (UID: "823e08fd-1d18-45dc-89ca-7a4212f46c38"). InnerVolumeSpecName "kube-api-access-gfvsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:00.199460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.199429    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfvsn\" (UniqueName: \"kubernetes.io/projected/823e08fd-1d18-45dc-89ca-7a4212f46c38-kube-api-access-gfvsn\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:36:00.199460 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.199457    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e08fd-1d18-45dc-89ca-7a4212f46c38-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:36:00.199655 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.199469    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/823e08fd-1d18-45dc-89ca-7a4212f46c38-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:36:00.727544 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.727505    2565 generic.go:356] "Generic (PLEG): container finished" podID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerID="e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707" exitCode=0
Jul 09 18:36:00.727959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.727552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"
Jul 09 18:36:00.727959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.727591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" event={"ID":"823e08fd-1d18-45dc-89ca-7a4212f46c38","Type":"ContainerDied","Data":"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"}
Jul 09 18:36:00.727959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.727643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd" event={"ID":"823e08fd-1d18-45dc-89ca-7a4212f46c38","Type":"ContainerDied","Data":"2aa24f0a4e6dc71fcc96c063ebe4c13bd444f3161f750929ca048813c03f47b3"}
Jul 09 18:36:00.727959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.727666    2565 scope.go:117] "RemoveContainer" containerID="e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"
Jul 09 18:36:00.736763 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.736746    2565 scope.go:117] "RemoveContainer" containerID="e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"
Jul 09 18:36:00.737056 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:36:00.737033    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707\": container with ID starting with e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707 not found: ID does not exist" containerID="e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"
Jul 09 18:36:00.737150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.737061    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707"} err="failed to get container status \"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707\": rpc error: code = NotFound desc = could not find container \"e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707\": container with ID starting with e0413074e70fc08721fa81de3f3fd2bf152f421a3a1342548700f997195c6707 not found: ID does not exist"
Jul 09 18:36:00.749740 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.749677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:36:00.754957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:00.754929    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche967xd"]
Jul 09 18:36:02.482729 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:02.482693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" path="/var/lib/kubelet/pods/823e08fd-1d18-45dc-89ca-7a4212f46c38/volumes"
Jul 09 18:36:28.472747 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.472712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473075    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c2464d-4ba6-42cd-9ba0-e21613124b48" containerName="istio-proxy"
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473086    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c2464d-4ba6-42cd-9ba0-e21613124b48" containerName="istio-proxy"
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473095    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerName="main"
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerName="main"
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473164    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="823e08fd-1d18-45dc-89ca-7a4212f46c38" containerName="main"
Jul 09 18:36:28.473221 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.473173    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c2464d-4ba6-42cd-9ba0-e21613124b48" containerName="istio-proxy"
Jul 09 18:36:28.478957 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.478933    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.481988 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.481969    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6fk4m\""
Jul 09 18:36:28.484195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.484173    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:36:28.485997 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.485776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 18:36:28.485997 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.485840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:36:28.486165 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.486026    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:36:28.493517 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.493496    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:36:28.538122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.538087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.538122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.538126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.538347 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.538150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlrg\" (UniqueName: \"kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.638755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.638713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.638938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.638780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.638938 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.638819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlrg\" (UniqueName: \"kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.639155 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.639137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.641544 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.641522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.660148 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.660118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlrg\" (UniqueName: \"kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.788415 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.788321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:28.933550 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:28.933527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:36:28.934339 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:36:28.934314    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode647d40f_8606_40b8_ad27_84fdf77d8699.slice/crio-1cadb47a99cb82a0e889801a650fdfc61e8bae9f989b97343cfca1a9afdffc26 WatchSource:0}: Error finding container 1cadb47a99cb82a0e889801a650fdfc61e8bae9f989b97343cfca1a9afdffc26: Status 404 returned error can't find the container with id 1cadb47a99cb82a0e889801a650fdfc61e8bae9f989b97343cfca1a9afdffc26
Jul 09 18:36:29.882152 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:29.882113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" event={"ID":"e647d40f-8606-40b8-ad27-84fdf77d8699","Type":"ContainerStarted","Data":"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"}
Jul 09 18:36:29.882599 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:29.882169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" event={"ID":"e647d40f-8606-40b8-ad27-84fdf77d8699","Type":"ContainerStarted","Data":"1cadb47a99cb82a0e889801a650fdfc61e8bae9f989b97343cfca1a9afdffc26"}
Jul 09 18:36:29.882599 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:29.882286    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:36:29.920604 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:36:29.920551    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" podStartSLOduration=1.920533688 podStartE2EDuration="1.920533688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:29.917316896 +0000 UTC m=+2012.004566199" watchObservedRunningTime="2026-07-09 18:36:29.920533688 +0000 UTC m=+2012.007782992"
Jul 09 18:37:00.889780 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:37:00.889750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:38:09.835807 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.835771    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:38:09.840340 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.840317    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:38:09.840625 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.840608    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:38:09.845006 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.844982    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:38:09.853607 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.853549    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:38:09.858818 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:38:09.858790    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:40:33.358853 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.358767    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:40:33.367677 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:40:33.367613    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:40:33.367677 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:33.367677 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:33.367677 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88" cmd=["/bin/sleep","15"]
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.367690    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" message="" err=<
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:40:33.367703    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerName="main" containerID="cri-o://a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"
Jul 09 18:40:33.367901 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.367735    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerName="main" containerID="cri-o://a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88" gracePeriod=60
Jul 09 18:40:33.622159 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.622100    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:40:33.674248 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.674198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjlrg\" (UniqueName: \"kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg\") pod \"e647d40f-8606-40b8-ad27-84fdf77d8699\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") "
Jul 09 18:40:33.674424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.674287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds\") pod \"e647d40f-8606-40b8-ad27-84fdf77d8699\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") "
Jul 09 18:40:33.674424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.674334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs\") pod \"e647d40f-8606-40b8-ad27-84fdf77d8699\" (UID: \"e647d40f-8606-40b8-ad27-84fdf77d8699\") "
Jul 09 18:40:33.674587 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.674549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e647d40f-8606-40b8-ad27-84fdf77d8699" (UID: "e647d40f-8606-40b8-ad27-84fdf77d8699"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:33.674721 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.674667    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e647d40f-8606-40b8-ad27-84fdf77d8699-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:40:33.676348 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.676326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg" (OuterVolumeSpecName: "kube-api-access-xjlrg") pod "e647d40f-8606-40b8-ad27-84fdf77d8699" (UID: "e647d40f-8606-40b8-ad27-84fdf77d8699"). InnerVolumeSpecName "kube-api-access-xjlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:33.676484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.676462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e647d40f-8606-40b8-ad27-84fdf77d8699" (UID: "e647d40f-8606-40b8-ad27-84fdf77d8699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:33.775823 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.775787    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjlrg\" (UniqueName: \"kubernetes.io/projected/e647d40f-8606-40b8-ad27-84fdf77d8699-kube-api-access-xjlrg\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:40:33.775823 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:33.775815    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e647d40f-8606-40b8-ad27-84fdf77d8699-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:40:34.167550 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.167520    2565 generic.go:356] "Generic (PLEG): container finished" podID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerID="a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88" exitCode=0
Jul 09 18:40:34.167737 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.167573    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"
Jul 09 18:40:34.167737 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.167588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" event={"ID":"e647d40f-8606-40b8-ad27-84fdf77d8699","Type":"ContainerDied","Data":"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"}
Jul 09 18:40:34.167737 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.167633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq" event={"ID":"e647d40f-8606-40b8-ad27-84fdf77d8699","Type":"ContainerDied","Data":"1cadb47a99cb82a0e889801a650fdfc61e8bae9f989b97343cfca1a9afdffc26"}
Jul 09 18:40:34.167737 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.167659    2565 scope.go:117] "RemoveContainer" containerID="a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"
Jul 09 18:40:34.177165 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.177147    2565 scope.go:117] "RemoveContainer" containerID="a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"
Jul 09 18:40:34.177435 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:40:34.177418    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88\": container with ID starting with a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88 not found: ID does not exist" containerID="a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"
Jul 09 18:40:34.177494 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.177443    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88"} err="failed to get container status \"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88\": rpc error: code = NotFound desc = could not find container \"a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88\": container with ID starting with a8dacd63002bf666545d92f121ae636c05c304239574e17087e65cb1317bec88 not found: ID does not exist"
Jul 09 18:40:34.215785 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.215747    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:40:34.217920 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.217898    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schewqhcq"]
Jul 09 18:40:34.482493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:34.482460    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" path="/var/lib/kubelet/pods/e647d40f-8606-40b8-ad27-84fdf77d8699/volumes"
Jul 09 18:40:50.331669 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.331635    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:40:50.332307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.332042    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerName="main"
Jul 09 18:40:50.332307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.332058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerName="main"
Jul 09 18:40:50.332307 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.332135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e647d40f-8606-40b8-ad27-84fdf77d8699" containerName="main"
Jul 09 18:40:50.337030 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.337004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.339870 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.339847    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:40:50.340960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.340659    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:40:50.340960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.340698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:40:50.340960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.340717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:40:50.340960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.340661    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:40:50.340960 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.340780    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-scqs6\""
Jul 09 18:40:50.347899 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.347879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:40:50.426127 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.426095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.426321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.426140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.426321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.426212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.426321 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.426293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgtr\" (UniqueName: \"kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.526908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.526860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.526908 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.526910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.527186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.526952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.527186 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.527009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgtr\" (UniqueName: \"kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.527645 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.527373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.527645 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.527619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.529610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.529585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.536787 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.536753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgtr\" (UniqueName: \"kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.649558 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.649476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:50.788582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:50.788556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:40:50.790620 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:40:50.790579    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820483b7_4c3c_4b85_9802_78949a93bb68.slice/crio-47c259d1140b77a714927c0eae5bf7b3cd590171ccf0bbfbe579e88ded456888 WatchSource:0}: Error finding container 47c259d1140b77a714927c0eae5bf7b3cd590171ccf0bbfbe579e88ded456888: Status 404 returned error can't find the container with id 47c259d1140b77a714927c0eae5bf7b3cd590171ccf0bbfbe579e88ded456888
Jul 09 18:40:51.268618 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:51.268576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerStarted","Data":"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"}
Jul 09 18:40:51.268768 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:51.268638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerStarted","Data":"47c259d1140b77a714927c0eae5bf7b3cd590171ccf0bbfbe579e88ded456888"}
Jul 09 18:40:52.274969 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:52.274938    2565 generic.go:356] "Generic (PLEG): container finished" podID="820483b7-4c3c-4b85-9802-78949a93bb68" containerID="9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4" exitCode=0
Jul 09 18:40:52.275366 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:52.275018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerDied","Data":"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"}
Jul 09 18:40:53.287035 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:53.287000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerStarted","Data":"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"}
Jul 09 18:40:53.287467 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:53.287144    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:40:53.313744 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:40:53.313693    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" podStartSLOduration=2.651432084 podStartE2EDuration="3.313677855s" totalImagesPullingTime="0s" totalInitContainerRuntime="662.245771ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:40:53.312374463 +0000 UTC m=+2275.399623767" watchObservedRunningTime="2026-07-09 18:40:53.313677855 +0000 UTC m=+2275.400927159"
Jul 09 18:41:24.294343 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:41:24.294308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:43:09.881225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.881113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:43:09.885729 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.885699    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:43:09.889624 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.889604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:43:09.892498 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.892475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:43:09.894289 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.894272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:43:09.900360 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:43:09.900342    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:44:16.274905 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.274871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:44:16.283927 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:44:16.283888    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:44:16.283927 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:16.283927 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:16.283927 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9" cmd=["/bin/sleep","15"]
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.283938    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" message="" err=<
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:44:16.283950    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="main" containerID="cri-o://ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"
Jul 09 18:44:16.284169 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.283996    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="main" containerID="cri-o://ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9" gracePeriod=60
Jul 09 18:44:16.539508 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.539449    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:44:16.648032 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds\") pod \"820483b7-4c3c-4b85-9802-78949a93bb68\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") "
Jul 09 18:44:16.648032 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpgtr\" (UniqueName: \"kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr\") pod \"820483b7-4c3c-4b85-9802-78949a93bb68\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") "
Jul 09 18:44:16.648291 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location\") pod \"820483b7-4c3c-4b85-9802-78949a93bb68\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") "
Jul 09 18:44:16.648291 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs\") pod \"820483b7-4c3c-4b85-9802-78949a93bb68\" (UID: \"820483b7-4c3c-4b85-9802-78949a93bb68\") "
Jul 09 18:44:16.648415 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648361    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "820483b7-4c3c-4b85-9802-78949a93bb68" (UID: "820483b7-4c3c-4b85-9802-78949a93bb68"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:16.648779 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.648756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "820483b7-4c3c-4b85-9802-78949a93bb68" (UID: "820483b7-4c3c-4b85-9802-78949a93bb68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:16.650208 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.650186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr" (OuterVolumeSpecName: "kube-api-access-wpgtr") pod "820483b7-4c3c-4b85-9802-78949a93bb68" (UID: "820483b7-4c3c-4b85-9802-78949a93bb68"). InnerVolumeSpecName "kube-api-access-wpgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:16.650293 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.650258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "820483b7-4c3c-4b85-9802-78949a93bb68" (UID: "820483b7-4c3c-4b85-9802-78949a93bb68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:16.748752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.748713    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:16.748752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.748746    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgtr\" (UniqueName: \"kubernetes.io/projected/820483b7-4c3c-4b85-9802-78949a93bb68-kube-api-access-wpgtr\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:16.748752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.748756    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820483b7-4c3c-4b85-9802-78949a93bb68-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:16.749016 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:16.748766    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820483b7-4c3c-4b85-9802-78949a93bb68-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:17.378057 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.378023    2565 generic.go:356] "Generic (PLEG): container finished" podID="820483b7-4c3c-4b85-9802-78949a93bb68" containerID="ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9" exitCode=0
Jul 09 18:44:17.378492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.378094    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"
Jul 09 18:44:17.378492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.378106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerDied","Data":"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"}
Jul 09 18:44:17.378492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.378187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4" event={"ID":"820483b7-4c3c-4b85-9802-78949a93bb68","Type":"ContainerDied","Data":"47c259d1140b77a714927c0eae5bf7b3cd590171ccf0bbfbe579e88ded456888"}
Jul 09 18:44:17.378492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.378208    2565 scope.go:117] "RemoveContainer" containerID="ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"
Jul 09 18:44:17.391252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.391204    2565 scope.go:117] "RemoveContainer" containerID="9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"
Jul 09 18:44:17.400071 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.400055    2565 scope.go:117] "RemoveContainer" containerID="ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"
Jul 09 18:44:17.400364 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:44:17.400347    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9\": container with ID starting with ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9 not found: ID does not exist" containerID="ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"
Jul 09 18:44:17.400424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.400373    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9"} err="failed to get container status \"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9\": rpc error: code = NotFound desc = could not find container \"ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9\": container with ID starting with ddae776d0ed8149ccc92f9c1fdea481b361149b2d18cf34f71759c31fcdf29d9 not found: ID does not exist"
Jul 09 18:44:17.400424 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.400390    2565 scope.go:117] "RemoveContainer" containerID="9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"
Jul 09 18:44:17.400668 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:44:17.400650    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4\": container with ID starting with 9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4 not found: ID does not exist" containerID="9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"
Jul 09 18:44:17.400728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.400677    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4"} err="failed to get container status \"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4\": rpc error: code = NotFound desc = could not find container \"9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4\": container with ID starting with 9ac74a6dd7015c71e0e046ee46a2b6781c9c835f2a6dcab70f793582bae909b4 not found: ID does not exist"
Jul 09 18:44:17.424682 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.424652    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:44:17.440420 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:17.440389    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-756d5mblz4"]
Jul 09 18:44:18.481902 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:18.481864    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" path="/var/lib/kubelet/pods/820483b7-4c3c-4b85-9802-78949a93bb68/volumes"
Jul 09 18:44:37.130362 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130326    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:37.130756 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130741    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="storage-initializer"
Jul 09 18:44:37.130803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130758    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="storage-initializer"
Jul 09 18:44:37.130803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130775    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="main"
Jul 09 18:44:37.130803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130782    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="main"
Jul 09 18:44:37.130897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.130850    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="820483b7-4c3c-4b85-9802-78949a93bb68" containerName="main"
Jul 09 18:44:37.133784 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.133764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.136483 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.136455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-jgwpq\""
Jul 09 18:44:37.136596 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.136457    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:44:37.136813 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.136799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:44:37.137226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.137213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:44:37.146503 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.146482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:37.219941 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.219897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.219952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.219978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220131 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220361 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvp5\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220361 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.220361 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.220204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321056 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvp5\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321354 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321736 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321796 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321853 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321853 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.321981 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.321963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.323588 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.323561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.323890 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.323868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.330929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.330905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvp5\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.331063 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.331017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-98n7v\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.444253 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.444196    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:37.591555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.591531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:37.593401 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:44:37.593375    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c2e0e7_3ff4_4ef0_8ee9_e39bf6546ca1.slice/crio-779490aa1850eb30f9c87a28fee2f69f8acbc942ee5fb6f7fb4c2e049eb761d4 WatchSource:0}: Error finding container 779490aa1850eb30f9c87a28fee2f69f8acbc942ee5fb6f7fb4c2e049eb761d4: Status 404 returned error can't find the container with id 779490aa1850eb30f9c87a28fee2f69f8acbc942ee5fb6f7fb4c2e049eb761d4
Jul 09 18:44:37.595489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.595458    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:37.595550 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.595525    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:37.595590 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:37.595556    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:38.488827 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:38.488793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" event={"ID":"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1","Type":"ContainerStarted","Data":"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"}
Jul 09 18:44:38.489255 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:38.488841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" event={"ID":"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1","Type":"ContainerStarted","Data":"779490aa1850eb30f9c87a28fee2f69f8acbc942ee5fb6f7fb4c2e049eb761d4"}
Jul 09 18:44:38.512168 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:38.512120    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" podStartSLOduration=1.512103963 podStartE2EDuration="1.512103963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:44:38.509455083 +0000 UTC m=+2500.596704399" watchObservedRunningTime="2026-07-09 18:44:38.512103963 +0000 UTC m=+2500.599353267"
Jul 09 18:44:39.444396 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:39.444355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:39.449478 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:39.449452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:39.494223 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:39.494192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:39.495224 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:39.495200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:40.239951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.239920    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"]
Jul 09 18:44:40.243439 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.243417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.259251 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.259208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"]
Jul 09 18:44:40.345383 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345383 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772mp\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-kube-api-access-772mp\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.345724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.345694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446203 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446203 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446496 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772mp\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-kube-api-access-772mp\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.446929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.446909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.447029 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.447019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.447184 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.447165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.448787 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.448759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.448982 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.448966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.454605 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.454571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772mp\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-kube-api-access-772mp\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.454757 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.454660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5f8378-56ff-4c15-b881-d6d7b5ffb8b9-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-m4ddf\" (UID: \"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.553861 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.553769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:40.705996 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.705969    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"]
Jul 09 18:44:40.707838 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:44:40.707810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5f8378_56ff_4c15_b881_d6d7b5ffb8b9.slice/crio-23a263251762c416094022e0074a5b227a6c9cf556503f670e553cd1d6830f53 WatchSource:0}: Error finding container 23a263251762c416094022e0074a5b227a6c9cf556503f670e553cd1d6830f53: Status 404 returned error can't find the container with id 23a263251762c416094022e0074a5b227a6c9cf556503f670e553cd1d6830f53
Jul 09 18:44:40.709945 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.709914    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:40.710046 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.709999    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:40.710109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:40.710072    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:41.505734 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:41.505697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" event={"ID":"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9","Type":"ContainerStarted","Data":"3f1ec6decc174656defbb5f452de6a5eda4651205c6d836936c2d71bc046a859"}
Jul 09 18:44:41.506027 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:41.505750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" event={"ID":"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9","Type":"ContainerStarted","Data":"23a263251762c416094022e0074a5b227a6c9cf556503f670e553cd1d6830f53"}
Jul 09 18:44:41.528948 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:41.528893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" podStartSLOduration=1.528879563 podStartE2EDuration="1.528879563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:44:41.52664954 +0000 UTC m=+2503.613898846" watchObservedRunningTime="2026-07-09 18:44:41.528879563 +0000 UTC m=+2503.616128866"
Jul 09 18:44:41.554949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:41.554912    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:41.559830 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:41.559806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:42.510777 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:42.510744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:42.511814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:42.511794    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:42.570091 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:42.570065    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:42.570479 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:42.570244    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" podUID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" containerName="istio-proxy" containerID="cri-o://2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080" gracePeriod=30
Jul 09 18:44:47.864928 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.864903    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:47.911952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.911848    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.911952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.911892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plvp5\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.911952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.911914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.911952 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.911938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.911997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912056    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912081    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912160    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo\") pod \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\" (UID: \"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1\") "
Jul 09 18:44:47.912334 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:47.912664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912370    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:47.912664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:47.912664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:47.912664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912533    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:47.912664 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912552    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:47.912929 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.912709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:44:47.941962 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.941922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:44:47.944744 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.944698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:47.944866 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.944747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5" (OuterVolumeSpecName: "kube-api-access-plvp5") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "kube-api-access-plvp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:47.944866 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:47.944816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" (UID: "c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:48.013297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013259    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013289    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013297 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013301    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013538 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013312    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013538 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013324    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013538 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013334    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.013538 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.013344    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvp5\" (UniqueName: \"kubernetes.io/projected/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1-kube-api-access-plvp5\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:44:48.556547 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.556510    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" containerID="2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080" exitCode=0
Jul 09 18:44:48.556755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.556565    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"
Jul 09 18:44:48.556755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.556596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" event={"ID":"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1","Type":"ContainerDied","Data":"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"}
Jul 09 18:44:48.556755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.556650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v" event={"ID":"c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1","Type":"ContainerDied","Data":"779490aa1850eb30f9c87a28fee2f69f8acbc942ee5fb6f7fb4c2e049eb761d4"}
Jul 09 18:44:48.556755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.556673    2565 scope.go:117] "RemoveContainer" containerID="2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"
Jul 09 18:44:48.565919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.565900    2565 scope.go:117] "RemoveContainer" containerID="2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"
Jul 09 18:44:48.566254 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:44:48.566212    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080\": container with ID starting with 2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080 not found: ID does not exist" containerID="2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"
Jul 09 18:44:48.566368 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.566267    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080"} err="failed to get container status \"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080\": rpc error: code = NotFound desc = could not find container \"2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080\": container with ID starting with 2f9e11dec882ac3a031e8d285d2ab67793aa9a605a680574a173ecf02173f080 not found: ID does not exist"
Jul 09 18:44:48.578183 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.578157    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:48.583098 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:48.583077    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-98n7v"]
Jul 09 18:44:50.482226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:50.482140    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" path="/var/lib/kubelet/pods/c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1/volumes"
Jul 09 18:44:52.581509 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.581474    2565 generic.go:356] "Generic (PLEG): container finished" podID="da5f8378-56ff-4c15-b881-d6d7b5ffb8b9" containerID="3f1ec6decc174656defbb5f452de6a5eda4651205c6d836936c2d71bc046a859" exitCode=0
Jul 09 18:44:52.581951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.581545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" event={"ID":"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9","Type":"ContainerDied","Data":"3f1ec6decc174656defbb5f452de6a5eda4651205c6d836936c2d71bc046a859"}
Jul 09 18:44:52.581951 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.581922    2565 scope.go:117] "RemoveContainer" containerID="3f1ec6decc174656defbb5f452de6a5eda4651205c6d836936c2d71bc046a859"
Jul 09 18:44:52.582903 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.582874    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:52.582999 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.582960    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:52.583062 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:52.583012    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 18:44:53.589086 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:53.589049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" event={"ID":"da5f8378-56ff-4c15-b881-d6d7b5ffb8b9","Type":"ContainerStarted","Data":"9e6273a2d384eefa50ffdd34f83ad6838c31881b5c241436a21def3a67436102"}
Jul 09 18:44:54.554317 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:54.554280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:54.555616 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:54.555582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" podUID="da5f8378-56ff-4c15-b881-d6d7b5ffb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 09 18:44:55.555189 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:55.555144    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" podUID="da5f8378-56ff-4c15-b881-d6d7b5ffb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 09 18:44:56.554445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:56.554403    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" podUID="da5f8378-56ff-4c15-b881-d6d7b5ffb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 09 18:44:57.511097 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:57.511045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:57.554892 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:57.554845    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf" podUID="da5f8378-56ff-4c15-b881-d6d7b5ffb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 09 18:44:58.559133 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:58.559103    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:44:58.560260 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:44:58.560219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-m4ddf"
Jul 09 18:45:04.463311 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.463276    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:45:04.463696 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.463634    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" containerName="istio-proxy"
Jul 09 18:45:04.463696 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.463645    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" containerName="istio-proxy"
Jul 09 18:45:04.463774 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.463729    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c2e0e7-3ff4-4ef0-8ee9-e39bf6546ca1" containerName="istio-proxy"
Jul 09 18:45:04.466977 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.466958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.472619 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.472598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-mhr2g\""
Jul 09 18:45:04.472854 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.472836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:45:04.472948 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.472838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:45:04.473735 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.473638    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:45:04.553413 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.553383    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:45:04.558348 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.558324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmzd\" (UniqueName: \"kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.558472 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.558373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.558517 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.558471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.558561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.558520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.659349 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.659317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.659532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.659362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.659532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.659387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.659532 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.659431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmzd\" (UniqueName: \"kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.660212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.660081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.660212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.660126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.668240 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.668213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.670784 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.670765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmzd\" (UniqueName: \"kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd\") pod \"stop-feature-test-kserve-router-scheduler-55ff964456-n74kh\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.776743 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.776651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:04.922582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:04.922556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:45:04.924219 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:45:04.924193    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0949110_0048_444f_b971_99bcb571b1b9.slice/crio-5983bfca1ed8c680d9698a57ec591c49355c3d7d21a46538e7022dd7aef854d7 WatchSource:0}: Error finding container 5983bfca1ed8c680d9698a57ec591c49355c3d7d21a46538e7022dd7aef854d7: Status 404 returned error can't find the container with id 5983bfca1ed8c680d9698a57ec591c49355c3d7d21a46538e7022dd7aef854d7
Jul 09 18:45:05.661140 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:05.661106    2565 generic.go:356] "Generic (PLEG): container finished" podID="e0949110-0048-444f-b971-99bcb571b1b9" containerID="e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f" exitCode=0
Jul 09 18:45:05.661581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:05.661188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" event={"ID":"e0949110-0048-444f-b971-99bcb571b1b9","Type":"ContainerDied","Data":"e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f"}
Jul 09 18:45:05.661581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:05.661263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" event={"ID":"e0949110-0048-444f-b971-99bcb571b1b9","Type":"ContainerStarted","Data":"5983bfca1ed8c680d9698a57ec591c49355c3d7d21a46538e7022dd7aef854d7"}
Jul 09 18:45:06.668589 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:06.668553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" event={"ID":"e0949110-0048-444f-b971-99bcb571b1b9","Type":"ContainerStarted","Data":"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"}
Jul 09 18:45:06.669034 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:06.668652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:45:06.690246 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:06.690186    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" podStartSLOduration=2.043071534 podStartE2EDuration="2.690168809s" totalImagesPullingTime="0s" totalInitContainerRuntime="647.097275ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:06.688210181 +0000 UTC m=+2528.775459484" watchObservedRunningTime="2026-07-09 18:45:06.690168809 +0000 UTC m=+2528.777418112"
Jul 09 18:45:37.676150 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:45:37.676120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:47:45.761280 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:45.761242    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:47:45.771492 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:47:45.771450    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:47:45.771492 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:47:45.771492 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:47:45.771492 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92" cmd=["/bin/sleep","15"]
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:45.771508    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" message="" err=<
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:47:45.771521    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="main" containerID="cri-o://b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"
Jul 09 18:47:45.771803 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:45.771570    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="main" containerID="cri-o://b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92" gracePeriod=60
Jul 09 18:47:46.053990 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.053963    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:47:46.225114 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds\") pod \"e0949110-0048-444f-b971-99bcb571b1b9\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") "
Jul 09 18:47:46.225299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs\") pod \"e0949110-0048-444f-b971-99bcb571b1b9\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") "
Jul 09 18:47:46.225299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fmzd\" (UniqueName: \"kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd\") pod \"e0949110-0048-444f-b971-99bcb571b1b9\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") "
Jul 09 18:47:46.225299 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location\") pod \"e0949110-0048-444f-b971-99bcb571b1b9\" (UID: \"e0949110-0048-444f-b971-99bcb571b1b9\") "
Jul 09 18:47:46.225483 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e0949110-0048-444f-b971-99bcb571b1b9" (UID: "e0949110-0048-444f-b971-99bcb571b1b9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:47:46.225551 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225483    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:47:46.225959 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.225932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0949110-0048-444f-b971-99bcb571b1b9" (UID: "e0949110-0048-444f-b971-99bcb571b1b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:47:46.227571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.227550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd" (OuterVolumeSpecName: "kube-api-access-7fmzd") pod "e0949110-0048-444f-b971-99bcb571b1b9" (UID: "e0949110-0048-444f-b971-99bcb571b1b9"). InnerVolumeSpecName "kube-api-access-7fmzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:47:46.227656 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.227552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0949110-0048-444f-b971-99bcb571b1b9" (UID: "e0949110-0048-444f-b971-99bcb571b1b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:47:46.326252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.326149    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0949110-0048-444f-b971-99bcb571b1b9-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:47:46.326252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.326178    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmzd\" (UniqueName: \"kubernetes.io/projected/e0949110-0048-444f-b971-99bcb571b1b9-kube-api-access-7fmzd\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:47:46.326252 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.326188    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0949110-0048-444f-b971-99bcb571b1b9-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:47:46.533840 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.533805    2565 generic.go:356] "Generic (PLEG): container finished" podID="e0949110-0048-444f-b971-99bcb571b1b9" containerID="b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92" exitCode=0
Jul 09 18:47:46.534018 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.533857    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"
Jul 09 18:47:46.534018 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.533885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" event={"ID":"e0949110-0048-444f-b971-99bcb571b1b9","Type":"ContainerDied","Data":"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"}
Jul 09 18:47:46.534018 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.533940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh" event={"ID":"e0949110-0048-444f-b971-99bcb571b1b9","Type":"ContainerDied","Data":"5983bfca1ed8c680d9698a57ec591c49355c3d7d21a46538e7022dd7aef854d7"}
Jul 09 18:47:46.534018 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.533970    2565 scope.go:117] "RemoveContainer" containerID="b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"
Jul 09 18:47:46.543105 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.543085    2565 scope.go:117] "RemoveContainer" containerID="e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f"
Jul 09 18:47:46.551475 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.551433    2565 scope.go:117] "RemoveContainer" containerID="b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"
Jul 09 18:47:46.551725 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:47:46.551700    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92\": container with ID starting with b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92 not found: ID does not exist" containerID="b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"
Jul 09 18:47:46.551788 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.551738    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92"} err="failed to get container status \"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92\": rpc error: code = NotFound desc = could not find container \"b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92\": container with ID starting with b0250ccab4247db2e798d16cd37bfa585c12836e51acb9f4e1131cd8a11d1e92 not found: ID does not exist"
Jul 09 18:47:46.551788 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.551765    2565 scope.go:117] "RemoveContainer" containerID="e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f"
Jul 09 18:47:46.552049 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:47:46.552031    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f\": container with ID starting with e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f not found: ID does not exist" containerID="e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f"
Jul 09 18:47:46.552151 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.552055    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f"} err="failed to get container status \"e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f\": rpc error: code = NotFound desc = could not find container \"e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f\": container with ID starting with e133685b359873534633252a579917d7dd9b799acee08b18d3eecacfe66e802f not found: ID does not exist"
Jul 09 18:47:46.552821 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.552802    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:47:46.556746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:46.556717    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-55ff964456-n74kh"]
Jul 09 18:47:48.482925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:47:48.482889    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0949110-0048-444f-b971-99bcb571b1b9" path="/var/lib/kubelet/pods/e0949110-0048-444f-b971-99bcb571b1b9/volumes"
Jul 09 18:48:09.918037 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.917951    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:48:09.922628 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.922602    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:48:09.927021 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.926995    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:48:09.936948 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.936925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:48:09.939582 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.939562    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:48:09.945644 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:48:09.945627    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:51:53.760755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.760713    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:51:53.761346 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.761322    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="main"
Jul 09 18:51:53.761441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.761348    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="main"
Jul 09 18:51:53.761441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.761369    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="storage-initializer"
Jul 09 18:51:53.761441 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.761379    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="storage-initializer"
Jul 09 18:51:53.761618 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.761491    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0949110-0048-444f-b971-99bcb571b1b9" containerName="main"
Jul 09 18:51:53.765195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.765172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.769647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.769460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-z8lmr\""
Jul 09 18:51:53.769647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.769480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:51:53.769647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.769557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:51:53.769647 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.769560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:51:53.778937 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.778911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:51:53.817176 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.817138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.817383 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.817207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcl8\" (UniqueName: \"kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.817383 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.817347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.817511 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.817382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918104 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918104 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918396 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918396 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcl8\" (UniqueName: \"kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918618 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.918672 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.918650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.920620 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.920602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:53.926416 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:53.926393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcl8\" (UniqueName: \"kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:54.077583 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:54.077492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:54.233148 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:54.233074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:51:54.234045 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:51:54.234015    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d85585e_866e_4113_b263_423dbf24359b.slice/crio-ae7ecbf650aa1f05288be627220b4624c7e4b2e4aad49cf2e5aaabf9e8a9aa08 WatchSource:0}: Error finding container ae7ecbf650aa1f05288be627220b4624c7e4b2e4aad49cf2e5aaabf9e8a9aa08: Status 404 returned error can't find the container with id ae7ecbf650aa1f05288be627220b4624c7e4b2e4aad49cf2e5aaabf9e8a9aa08
Jul 09 18:51:54.900777 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:54.900740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerStarted","Data":"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"}
Jul 09 18:51:54.901225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:54.900800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerStarted","Data":"ae7ecbf650aa1f05288be627220b4624c7e4b2e4aad49cf2e5aaabf9e8a9aa08"}
Jul 09 18:51:55.909111 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:55.909076    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d85585e-866e-4113-b263-423dbf24359b" containerID="2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757" exitCode=0
Jul 09 18:51:55.909564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:55.909177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerDied","Data":"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"}
Jul 09 18:51:56.916612 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:56.916566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerStarted","Data":"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"}
Jul 09 18:51:56.917019 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:56.916641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:51:56.938355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:51:56.938305    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" podStartSLOduration=3.2448310989999998 podStartE2EDuration="3.938289077s" totalImagesPullingTime="0s" totalInitContainerRuntime="693.457978ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:51:56.937571658 +0000 UTC m=+2939.024820979" watchObservedRunningTime="2026-07-09 18:51:56.938289077 +0000 UTC m=+2939.025538383"
Jul 09 18:52:27.924259 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:52:27.924152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:53:09.961578 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.961550    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:53:09.966282 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.966259    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:53:09.972382 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.972356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:53:09.973292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.973272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:53:09.976688 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.976670    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:53:09.983791 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:09.983772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:53:48.764163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.764128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:53:48.767764 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.767739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.771454 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.771433    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-dhhld\""
Jul 09 18:53:48.771581 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.771436    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 18:53:48.778571 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.778547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzbf\" (UniqueName: \"kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.778665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.778579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.778665 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.778600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.778741 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.778720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.784742 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.784718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:53:48.879935 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.879903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzbf\" (UniqueName: \"kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.880107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.879939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.880107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.879962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.880107 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.880036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.880470 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.880448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.880555 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.880532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.882644 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.882625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:48.889328 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:48.889288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzbf\" (UniqueName: \"kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf\") pod \"tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:49.078661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:49.078561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:49.219943 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:49.219914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:53:49.221349 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:53:49.221318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bc75be_80e6_4257_ab0f_3303a7d2f33e.slice/crio-4b3c44e582d85ab7e6c1a37b15f7f0f35989c4a032184d7155d871ebb094cdaf WatchSource:0}: Error finding container 4b3c44e582d85ab7e6c1a37b15f7f0f35989c4a032184d7155d871ebb094cdaf: Status 404 returned error can't find the container with id 4b3c44e582d85ab7e6c1a37b15f7f0f35989c4a032184d7155d871ebb094cdaf
Jul 09 18:53:49.550224 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:49.550179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerStarted","Data":"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"}
Jul 09 18:53:49.550426 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:49.550278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerStarted","Data":"4b3c44e582d85ab7e6c1a37b15f7f0f35989c4a032184d7155d871ebb094cdaf"}
Jul 09 18:53:50.560489 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:50.560455    2565 generic.go:356] "Generic (PLEG): container finished" podID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerID="c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd" exitCode=0
Jul 09 18:53:50.560921 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:50.560545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerDied","Data":"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"}
Jul 09 18:53:51.568681 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:51.568648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerStarted","Data":"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"}
Jul 09 18:53:51.569083 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:51.568756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:53:51.592157 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:53:51.592106    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" podStartSLOduration=2.928501109 podStartE2EDuration="3.592090292s" totalImagesPullingTime="0s" totalInitContainerRuntime="663.589183ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:51.59010758 +0000 UTC m=+3053.677356884" watchObservedRunningTime="2026-07-09 18:53:51.592090292 +0000 UTC m=+3053.679339595"
Jul 09 18:54:22.575875 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:54:22.575842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:55:54.244919 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.244882    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:55:54.253864 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:55:54.253820    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:55:54.253864 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:55:54.253864 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:55:54.253864 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8" cmd=["/bin/sleep","15"]
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.254205    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" message="" err=<
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:55:54.254251    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:55:54.254258 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="main" containerID="cri-o://6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"
Jul 09 18:55:54.254623 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.254294    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="main" containerID="cri-o://6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8" gracePeriod=60
Jul 09 18:55:54.518824 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.518798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:55:54.609607 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.609574    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs\") pod \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") "
Jul 09 18:55:54.609607 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.609612    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds\") pod \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") "
Jul 09 18:55:54.609873 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.609711    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttzbf\" (UniqueName: \"kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf\") pod \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") "
Jul 09 18:55:54.609873 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.609787    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location\") pod \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\" (UID: \"97bc75be-80e6-4257-ab0f-3303a7d2f33e\") "
Jul 09 18:55:54.609988 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.609929    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "97bc75be-80e6-4257-ab0f-3303a7d2f33e" (UID: "97bc75be-80e6-4257-ab0f-3303a7d2f33e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:55:54.610128 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.610104    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:55:54.610511 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.610490    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97bc75be-80e6-4257-ab0f-3303a7d2f33e" (UID: "97bc75be-80e6-4257-ab0f-3303a7d2f33e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:55:54.611925 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.611896    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97bc75be-80e6-4257-ab0f-3303a7d2f33e" (UID: "97bc75be-80e6-4257-ab0f-3303a7d2f33e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:54.612281 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.612225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf" (OuterVolumeSpecName: "kube-api-access-ttzbf") pod "97bc75be-80e6-4257-ab0f-3303a7d2f33e" (UID: "97bc75be-80e6-4257-ab0f-3303a7d2f33e"). InnerVolumeSpecName "kube-api-access-ttzbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:54.711556 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.711518    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzbf\" (UniqueName: \"kubernetes.io/projected/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kube-api-access-ttzbf\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:55:54.711556 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.711551    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97bc75be-80e6-4257-ab0f-3303a7d2f33e-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:55:54.711556 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:54.711561    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc75be-80e6-4257-ab0f-3303a7d2f33e-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:55:55.228272 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.228214    2565 generic.go:356] "Generic (PLEG): container finished" podID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerID="6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8" exitCode=0
Jul 09 18:55:55.228445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.228279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerDied","Data":"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"}
Jul 09 18:55:55.228445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.228319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj" event={"ID":"97bc75be-80e6-4257-ab0f-3303a7d2f33e","Type":"ContainerDied","Data":"4b3c44e582d85ab7e6c1a37b15f7f0f35989c4a032184d7155d871ebb094cdaf"}
Jul 09 18:55:55.228445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.228340    2565 scope.go:117] "RemoveContainer" containerID="6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"
Jul 09 18:55:55.228445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.228285    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"
Jul 09 18:55:55.238043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.238019    2565 scope.go:117] "RemoveContainer" containerID="c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"
Jul 09 18:55:55.246979 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.246956    2565 scope.go:117] "RemoveContainer" containerID="6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"
Jul 09 18:55:55.247278 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:55:55.247247    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8\": container with ID starting with 6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8 not found: ID does not exist" containerID="6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"
Jul 09 18:55:55.247327 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.247277    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8"} err="failed to get container status \"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8\": rpc error: code = NotFound desc = could not find container \"6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8\": container with ID starting with 6f192a576588e9b40a276b223d2cab44ced3c2b1aede2b75d9a4cc728ea2ddd8 not found: ID does not exist"
Jul 09 18:55:55.247327 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.247298    2565 scope.go:117] "RemoveContainer" containerID="c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"
Jul 09 18:55:55.247587 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:55:55.247560    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd\": container with ID starting with c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd not found: ID does not exist" containerID="c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"
Jul 09 18:55:55.247686 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.247594    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd"} err="failed to get container status \"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd\": rpc error: code = NotFound desc = could not find container \"c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd\": container with ID starting with c316bac7bde50e79f963261d9a57d62a8921ce94b1443414514ac395e0eeecdd not found: ID does not exist"
Jul 09 18:55:55.251377 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.251354    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:55:55.255813 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:55.255792    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-88bdffc4d-m27nj"]
Jul 09 18:55:56.482206 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:55:56.482167    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" path="/var/lib/kubelet/pods/97bc75be-80e6-4257-ab0f-3303a7d2f33e/volumes"
Jul 09 18:56:00.296773 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.296739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:56:00.307305 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:00.307267    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:56:00.307305 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:56:00.307305 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:56:00.307305 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7" cmd=["/bin/sleep","15"]
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.307330    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" message="" err=<
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:00.307347    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="main" containerID="cri-o://69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"
Jul 09 18:56:00.307628 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.307389    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="main" containerID="cri-o://69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7" gracePeriod=60
Jul 09 18:56:00.567041 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.567013    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:56:00.663049 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds\") pod \"3d85585e-866e-4113-b263-423dbf24359b\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") "
Jul 09 18:56:00.663292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location\") pod \"3d85585e-866e-4113-b263-423dbf24359b\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") "
Jul 09 18:56:00.663292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcl8\" (UniqueName: \"kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8\") pod \"3d85585e-866e-4113-b263-423dbf24359b\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") "
Jul 09 18:56:00.663292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663210    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs\") pod \"3d85585e-866e-4113-b263-423dbf24359b\" (UID: \"3d85585e-866e-4113-b263-423dbf24359b\") "
Jul 09 18:56:00.663492 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663400    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3d85585e-866e-4113-b263-423dbf24359b" (UID: "3d85585e-866e-4113-b263-423dbf24359b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:00.663606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663575    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:56:00.663904 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.663873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d85585e-866e-4113-b263-423dbf24359b" (UID: "3d85585e-866e-4113-b263-423dbf24359b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:00.665487 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.665454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d85585e-866e-4113-b263-423dbf24359b" (UID: "3d85585e-866e-4113-b263-423dbf24359b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:56:00.665612 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.665569    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8" (OuterVolumeSpecName: "kube-api-access-zpcl8") pod "3d85585e-866e-4113-b263-423dbf24359b" (UID: "3d85585e-866e-4113-b263-423dbf24359b"). InnerVolumeSpecName "kube-api-access-zpcl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:56:00.764746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.764708    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcl8\" (UniqueName: \"kubernetes.io/projected/3d85585e-866e-4113-b263-423dbf24359b-kube-api-access-zpcl8\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:56:00.764746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.764738    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d85585e-866e-4113-b263-423dbf24359b-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:56:00.764746 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:00.764749    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d85585e-866e-4113-b263-423dbf24359b-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:56:01.266163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.266132    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d85585e-866e-4113-b263-423dbf24359b" containerID="69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7" exitCode=0
Jul 09 18:56:01.266163 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.266175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerDied","Data":"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"}
Jul 09 18:56:01.266466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.266188    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"
Jul 09 18:56:01.266466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.266224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb" event={"ID":"3d85585e-866e-4113-b263-423dbf24359b","Type":"ContainerDied","Data":"ae7ecbf650aa1f05288be627220b4624c7e4b2e4aad49cf2e5aaabf9e8a9aa08"}
Jul 09 18:56:01.266466 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.266273    2565 scope.go:117] "RemoveContainer" containerID="69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"
Jul 09 18:56:01.275723 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.275706    2565 scope.go:117] "RemoveContainer" containerID="2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"
Jul 09 18:56:01.283749 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.283732    2565 scope.go:117] "RemoveContainer" containerID="69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"
Jul 09 18:56:01.283994 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:01.283974    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7\": container with ID starting with 69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7 not found: ID does not exist" containerID="69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"
Jul 09 18:56:01.284043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.284003    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7"} err="failed to get container status \"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7\": rpc error: code = NotFound desc = could not find container \"69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7\": container with ID starting with 69eca3faede2d3ca801b529989a08d49726806a16be4f56a18beebcc14bb34e7 not found: ID does not exist"
Jul 09 18:56:01.284043 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.284020    2565 scope.go:117] "RemoveContainer" containerID="2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"
Jul 09 18:56:01.288363 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:01.287513    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757\": container with ID starting with 2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757 not found: ID does not exist" containerID="2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"
Jul 09 18:56:01.288363 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.287552    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757"} err="failed to get container status \"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757\": rpc error: code = NotFound desc = could not find container \"2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757\": container with ID starting with 2fc7cc6345ffcc1c6dab638b070d88d8200afc43a7978326a73c25707cabc757 not found: ID does not exist"
Jul 09 18:56:01.291833 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.291566    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:56:01.295518 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:01.295496    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheq5ghb"]
Jul 09 18:56:02.481807 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:02.481774    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d85585e-866e-4113-b263-423dbf24359b" path="/var/lib/kubelet/pods/3d85585e-866e-4113-b263-423dbf24359b/volumes"
Jul 09 18:56:27.183355 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183318    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:56:27.183896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183862    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="main"
Jul 09 18:56:27.183896 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183877    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="main"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="storage-initializer"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183921    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="storage-initializer"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="main"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183942    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="main"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183956    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="storage-initializer"
Jul 09 18:56:27.184040 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.183965    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="storage-initializer"
Jul 09 18:56:27.184329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.184084    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d85585e-866e-4113-b263-423dbf24359b" containerName="main"
Jul 09 18:56:27.184329 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.184099    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bc75be-80e6-4257-ab0f-3303a7d2f33e" containerName="main"
Jul 09 18:56:27.187766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.187746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.202075 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.202054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:56:27.202343 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:27.202320    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prestop-hook-test-epp-sa-dockercfg-km4gk\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-km4gk\"" type="*v1.Secret"
Jul 09 18:56:27.202897 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:27.202870    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.203017 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:27.202996    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Jul 09 18:56:27.203270 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:27.203214    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prestop-hook-test-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\"" type="*v1.Secret"
Jul 09 18:56:27.207760 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.207734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:56:27.281205 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.281169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.281412 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.281221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.281412 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.281294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.281412 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.281311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqql\" (UniqueName: \"kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.382677 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.382636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.382926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.382712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.382926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.382775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.382926 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.382803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqql\" (UniqueName: \"kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.383153 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.383124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.383225 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.383133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:27.397450 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:27.397425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqql\" (UniqueName: \"kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:28.073491 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.073456    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-km4gk\""
Jul 09 18:56:28.383170 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:28.383068    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Jul 09 18:56:28.383591 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:56:28.383188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs podName:c96f11fa-7a81-41e1-9805-438460f2049f nodeName:}" failed. No retries permitted until 2026-07-09 18:56:28.883170039 +0000 UTC m=+3210.970419325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs") pod "prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" (UID: "c96f11fa-7a81-41e1-9805-438460f2049f") : failed to sync secret cache: timed out waiting for the condition
Jul 09 18:56:28.523595 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.523562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 18:56:28.726695 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.726669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:56:28.899208 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.899176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:28.902083 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.902055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:28.998012 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:28.997912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:29.148102 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:29.148068    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:56:29.151324 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:56:29.151295    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96f11fa_7a81_41e1_9805_438460f2049f.slice/crio-55f315ffa520ee4c9af0ab27c88086fb26ccb8e696e89baa288eda3dd4ba98cf WatchSource:0}: Error finding container 55f315ffa520ee4c9af0ab27c88086fb26ccb8e696e89baa288eda3dd4ba98cf: Status 404 returned error can't find the container with id 55f315ffa520ee4c9af0ab27c88086fb26ccb8e696e89baa288eda3dd4ba98cf
Jul 09 18:56:29.420949 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:29.420850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerStarted","Data":"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"}
Jul 09 18:56:29.421350 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:29.420954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerStarted","Data":"55f315ffa520ee4c9af0ab27c88086fb26ccb8e696e89baa288eda3dd4ba98cf"}
Jul 09 18:56:30.428668 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:30.428633    2565 generic.go:356] "Generic (PLEG): container finished" podID="c96f11fa-7a81-41e1-9805-438460f2049f" containerID="ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5" exitCode=0
Jul 09 18:56:30.429079 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:30.428667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerDied","Data":"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"}
Jul 09 18:56:31.436271 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:31.436224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerStarted","Data":"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"}
Jul 09 18:56:31.436672 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:31.436374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:56:31.460109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:31.460059    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" podStartSLOduration=3.79810579 podStartE2EDuration="4.460043551s" totalImagesPullingTime="0s" totalInitContainerRuntime="661.937761ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:31.458656273 +0000 UTC m=+3213.545905590" watchObservedRunningTime="2026-07-09 18:56:31.460043551 +0000 UTC m=+3213.547292855"
Jul 09 18:56:56.653459 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.653418    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:56:56.657309 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.657285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.661812 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.661784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-nz7fp\""
Jul 09 18:56:56.661969 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.661835    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 09 18:56:56.671865 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.671834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:56:56.745564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.745531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.745760 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.745623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.745760 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.745663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9qj\" (UniqueName: \"kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.846447 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.846416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.846632 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.846466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9qj\" (UniqueName: \"kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.846632 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.846504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.846902 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.846877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.849011 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.848989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.855759 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.855737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9qj\" (UniqueName: \"kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:56.970109 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:56.970069    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:57.119561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:57.119407    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:56:57.121999 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:56:57.121971    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2973fc22_a2a3_4f88_8134_e004e854dcd4.slice/crio-41be449dd5289bbeb564c6dd6abebb18a2103fc93ea82c451c68912b9920656d WatchSource:0}: Error finding container 41be449dd5289bbeb564c6dd6abebb18a2103fc93ea82c451c68912b9920656d: Status 404 returned error can't find the container with id 41be449dd5289bbeb564c6dd6abebb18a2103fc93ea82c451c68912b9920656d
Jul 09 18:56:57.580313 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:57.580279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" event={"ID":"2973fc22-a2a3-4f88-8134-e004e854dcd4","Type":"ContainerStarted","Data":"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"}
Jul 09 18:56:57.580528 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:57.580326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" event={"ID":"2973fc22-a2a3-4f88-8134-e004e854dcd4","Type":"ContainerStarted","Data":"41be449dd5289bbeb564c6dd6abebb18a2103fc93ea82c451c68912b9920656d"}
Jul 09 18:56:57.580528 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:57.580398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:56:57.602737 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:56:57.602684    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" podStartSLOduration=1.602668789 podStartE2EDuration="1.602668789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:57.600654219 +0000 UTC m=+3239.687903526" watchObservedRunningTime="2026-07-09 18:56:57.602668789 +0000 UTC m=+3239.689918112"
Jul 09 18:57:02.442893 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:02.442862    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:57:28.587872 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:28.587845    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:57:30.223985 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.223947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:57:30.233218 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:57:30.233185    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:57:30.233218 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:57:30.233218 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:57:30.233218 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5" cmd=["/bin/sleep","15"]
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.233252    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" message="" err=<
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:57:30.233278    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerName="main" containerID="cri-o://199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"
Jul 09 18:57:30.233547 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.233329    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerName="main" containerID="cri-o://199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5" gracePeriod=60
Jul 09 18:57:30.499114 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.499078    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:57:30.559269 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.559208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs\") pod \"2973fc22-a2a3-4f88-8134-e004e854dcd4\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") "
Jul 09 18:57:30.559437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.559290    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx9qj\" (UniqueName: \"kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj\") pod \"2973fc22-a2a3-4f88-8134-e004e854dcd4\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") "
Jul 09 18:57:30.559437 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.559327    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds\") pod \"2973fc22-a2a3-4f88-8134-e004e854dcd4\" (UID: \"2973fc22-a2a3-4f88-8134-e004e854dcd4\") "
Jul 09 18:57:30.559810 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.559623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2973fc22-a2a3-4f88-8134-e004e854dcd4" (UID: "2973fc22-a2a3-4f88-8134-e004e854dcd4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:57:30.561473 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.561452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2973fc22-a2a3-4f88-8134-e004e854dcd4" (UID: "2973fc22-a2a3-4f88-8134-e004e854dcd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:57:30.561561 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.561527    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj" (OuterVolumeSpecName: "kube-api-access-gx9qj") pod "2973fc22-a2a3-4f88-8134-e004e854dcd4" (UID: "2973fc22-a2a3-4f88-8134-e004e854dcd4"). InnerVolumeSpecName "kube-api-access-gx9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:57:30.660791 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.660755    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2973fc22-a2a3-4f88-8134-e004e854dcd4-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:57:30.660791 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.660784    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9qj\" (UniqueName: \"kubernetes.io/projected/2973fc22-a2a3-4f88-8134-e004e854dcd4-kube-api-access-gx9qj\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:57:30.660791 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.660796    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2973fc22-a2a3-4f88-8134-e004e854dcd4-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:57:30.757457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.757356    2565 generic.go:356] "Generic (PLEG): container finished" podID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerID="199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5" exitCode=0
Jul 09 18:57:30.757457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.757398    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"
Jul 09 18:57:30.757457 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.757438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" event={"ID":"2973fc22-a2a3-4f88-8134-e004e854dcd4","Type":"ContainerDied","Data":"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"}
Jul 09 18:57:30.757714 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.757499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7" event={"ID":"2973fc22-a2a3-4f88-8134-e004e854dcd4","Type":"ContainerDied","Data":"41be449dd5289bbeb564c6dd6abebb18a2103fc93ea82c451c68912b9920656d"}
Jul 09 18:57:30.757714 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.757522    2565 scope.go:117] "RemoveContainer" containerID="199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"
Jul 09 18:57:30.767568 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.767550    2565 scope.go:117] "RemoveContainer" containerID="199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"
Jul 09 18:57:30.767839 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:57:30.767818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5\": container with ID starting with 199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5 not found: ID does not exist" containerID="199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"
Jul 09 18:57:30.767883 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.767851    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5"} err="failed to get container status \"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5\": rpc error: code = NotFound desc = could not find container \"199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5\": container with ID starting with 199c192a90e92d98f9c90d27baa0ccfb19b0b3187f30ec54a53855e7239184c5 not found: ID does not exist"
Jul 09 18:57:30.780001 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.779967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:57:30.787081 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:30.787058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfhlhw7"]
Jul 09 18:57:32.482503 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:57:32.482470    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" path="/var/lib/kubelet/pods/2973fc22-a2a3-4f88-8134-e004e854dcd4/volumes"
Jul 09 18:58:02.212865 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.212823    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:02.213436 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.213247    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerName="main"
Jul 09 18:58:02.213436 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.213264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerName="main"
Jul 09 18:58:02.213436 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.213357    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2973fc22-a2a3-4f88-8134-e004e854dcd4" containerName="main"
Jul 09 18:58:02.216733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.216708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.225352 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.225308    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-gx7fw\""
Jul 09 18:58:02.226178 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.226163    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 09 18:58:02.239708 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.239672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:02.351815 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.351771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.352020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.351894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.352020 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.351933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.452976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.452937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.452976 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.452983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.453275 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.453017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.453400 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.453381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.455564 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.455542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.461037 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.461015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.527653 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.527553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.670827 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.670802    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:02.672473 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:58:02.672447    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17f7a5d_5e11_4935_9799_da75fd7e24c0.slice/crio-b7fa20956a6ff6dda293942c8c2d5f64ecf13d0ce4b1d68cc268b88d20a6afeb WatchSource:0}: Error finding container b7fa20956a6ff6dda293942c8c2d5f64ecf13d0ce4b1d68cc268b88d20a6afeb: Status 404 returned error can't find the container with id b7fa20956a6ff6dda293942c8c2d5f64ecf13d0ce4b1d68cc268b88d20a6afeb
Jul 09 18:58:02.938595 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.938558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" event={"ID":"b17f7a5d-5e11-4935-9799-da75fd7e24c0","Type":"ContainerStarted","Data":"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"}
Jul 09 18:58:02.938766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.938614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" event={"ID":"b17f7a5d-5e11-4935-9799-da75fd7e24c0","Type":"ContainerStarted","Data":"b7fa20956a6ff6dda293942c8c2d5f64ecf13d0ce4b1d68cc268b88d20a6afeb"}
Jul 09 18:58:02.938766 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.938651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:02.961524 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:02.961474    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" podStartSLOduration=0.96145857 podStartE2EDuration="961.45857ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:02.957796864 +0000 UTC m=+3305.045046167" watchObservedRunningTime="2026-07-09 18:58:02.96145857 +0000 UTC m=+3305.048707873"
Jul 09 18:58:09.995886 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:09.995794    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:58:10.000445 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:10.000423    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:58:10.007211 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:10.007189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:58:10.009478 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:10.009451    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 18:58:10.013728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:10.013711    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 18:58:10.020655 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:10.020639    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 18:58:33.945811 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:33.945735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:45.538024 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.537986    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:45.547296 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:45.547257    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:58:45.547296 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:45.547296 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:45.547296 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45" cmd=["/bin/sleep","15"]
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.547319    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" message="" err=<
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:45.547334    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerName="main" containerID="cri-o://ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"
Jul 09 18:58:45.547606 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.547386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerName="main" containerID="cri-o://ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45" gracePeriod=60
Jul 09 18:58:45.803247 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.803213    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:45.933661 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.933627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz\") pod \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") "
Jul 09 18:58:45.933851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.933689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds\") pod \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") "
Jul 09 18:58:45.933851 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.933707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs\") pod \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\" (UID: \"b17f7a5d-5e11-4935-9799-da75fd7e24c0\") "
Jul 09 18:58:45.933987 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.933933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b17f7a5d-5e11-4935-9799-da75fd7e24c0" (UID: "b17f7a5d-5e11-4935-9799-da75fd7e24c0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:45.935844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.935806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b17f7a5d-5e11-4935-9799-da75fd7e24c0" (UID: "b17f7a5d-5e11-4935-9799-da75fd7e24c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:45.935844 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:45.935832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz" (OuterVolumeSpecName: "kube-api-access-2d7bz") pod "b17f7a5d-5e11-4935-9799-da75fd7e24c0" (UID: "b17f7a5d-5e11-4935-9799-da75fd7e24c0"). InnerVolumeSpecName "kube-api-access-2d7bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:58:46.034814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.034778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/b17f7a5d-5e11-4935-9799-da75fd7e24c0-kube-api-access-2d7bz\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.034814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.034808    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.034814 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.034820    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17f7a5d-5e11-4935-9799-da75fd7e24c0-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.183961 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.183847    2565 generic.go:356] "Generic (PLEG): container finished" podID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerID="ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45" exitCode=0
Jul 09 18:58:46.184268 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.183948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" event={"ID":"b17f7a5d-5e11-4935-9799-da75fd7e24c0","Type":"ContainerDied","Data":"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"}
Jul 09 18:58:46.184515 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.184487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg" event={"ID":"b17f7a5d-5e11-4935-9799-da75fd7e24c0","Type":"ContainerDied","Data":"b7fa20956a6ff6dda293942c8c2d5f64ecf13d0ce4b1d68cc268b88d20a6afeb"}
Jul 09 18:58:46.184614 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.184532    2565 scope.go:117] "RemoveContainer" containerID="ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"
Jul 09 18:58:46.185755 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.183968    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"
Jul 09 18:58:46.204610 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.204587    2565 scope.go:117] "RemoveContainer" containerID="ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"
Jul 09 18:58:46.204940 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:46.204918    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45\": container with ID starting with ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45 not found: ID does not exist" containerID="ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"
Jul 09 18:58:46.205014 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.204947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45"} err="failed to get container status \"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45\": rpc error: code = NotFound desc = could not find container \"ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45\": container with ID starting with ce13ed3de24b7b880d382fd1c22d843b889a66d8a610c3b130649f89aad90c45 not found: ID does not exist"
Jul 09 18:58:46.212470 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.212441    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:46.216837 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.216811    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche892mg"]
Jul 09 18:58:46.481913 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:46.481877    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" path="/var/lib/kubelet/pods/b17f7a5d-5e11-4935-9799-da75fd7e24c0/volumes"
Jul 09 18:58:48.659493 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:48.659451    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:58:48.668955 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:48.668922    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:58:48.668955 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:48.668955 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:48.668955 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1" cmd=["/bin/sleep","15"]
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:48.668970    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" message="" err=<
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:48.668983    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="main" containerID="cri-o://2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"
Jul 09 18:58:48.669195 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:48.669019    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="main" containerID="cri-o://2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1" gracePeriod=60
Jul 09 18:58:48.924048 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:48.923985    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:58:49.063072 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwqql\" (UniqueName: \"kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql\") pod \"c96f11fa-7a81-41e1-9805-438460f2049f\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") "
Jul 09 18:58:49.063287 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063083    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds\") pod \"c96f11fa-7a81-41e1-9805-438460f2049f\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") "
Jul 09 18:58:49.063287 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") pod \"c96f11fa-7a81-41e1-9805-438460f2049f\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") "
Jul 09 18:58:49.063287 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location\") pod \"c96f11fa-7a81-41e1-9805-438460f2049f\" (UID: \"c96f11fa-7a81-41e1-9805-438460f2049f\") "
Jul 09 18:58:49.063468 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c96f11fa-7a81-41e1-9805-438460f2049f" (UID: "c96f11fa-7a81-41e1-9805-438460f2049f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:49.063885 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.063863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c96f11fa-7a81-41e1-9805-438460f2049f" (UID: "c96f11fa-7a81-41e1-9805-438460f2049f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:49.065273 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.065220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c96f11fa-7a81-41e1-9805-438460f2049f" (UID: "c96f11fa-7a81-41e1-9805-438460f2049f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:49.065273 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.065253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql" (OuterVolumeSpecName: "kube-api-access-xwqql") pod "c96f11fa-7a81-41e1-9805-438460f2049f" (UID: "c96f11fa-7a81-41e1-9805-438460f2049f"). InnerVolumeSpecName "kube-api-access-xwqql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:58:49.164484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.164447    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqql\" (UniqueName: \"kubernetes.io/projected/c96f11fa-7a81-41e1-9805-438460f2049f-kube-api-access-xwqql\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.164484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.164481    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.164484 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.164495    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96f11fa-7a81-41e1-9805-438460f2049f-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.164733 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.164503    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c96f11fa-7a81-41e1-9805-438460f2049f-kserve-provision-location\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.203573 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.203539    2565 generic.go:356] "Generic (PLEG): container finished" podID="c96f11fa-7a81-41e1-9805-438460f2049f" containerID="2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1" exitCode=0
Jul 09 18:58:49.203727 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.203590    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"
Jul 09 18:58:49.203727 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.203626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerDied","Data":"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"}
Jul 09 18:58:49.203727 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.203685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v" event={"ID":"c96f11fa-7a81-41e1-9805-438460f2049f","Type":"ContainerDied","Data":"55f315ffa520ee4c9af0ab27c88086fb26ccb8e696e89baa288eda3dd4ba98cf"}
Jul 09 18:58:49.203727 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.203708    2565 scope.go:117] "RemoveContainer" containerID="2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"
Jul 09 18:58:49.213728 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.213713    2565 scope.go:117] "RemoveContainer" containerID="ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"
Jul 09 18:58:49.222064 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.222045    2565 scope.go:117] "RemoveContainer" containerID="2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"
Jul 09 18:58:49.222347 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:49.222325    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1\": container with ID starting with 2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1 not found: ID does not exist" containerID="2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"
Jul 09 18:58:49.222406 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.222356    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1"} err="failed to get container status \"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1\": rpc error: code = NotFound desc = could not find container \"2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1\": container with ID starting with 2384e29596eb31cccb2f6b417a0414cd562f25b70cdddaf11714097f86d07fd1 not found: ID does not exist"
Jul 09 18:58:49.222406 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.222373    2565 scope.go:117] "RemoveContainer" containerID="ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"
Jul 09 18:58:49.222633 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:58:49.222615    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5\": container with ID starting with ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5 not found: ID does not exist" containerID="ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"
Jul 09 18:58:49.222684 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.222638    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5"} err="failed to get container status \"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5\": rpc error: code = NotFound desc = could not find container \"ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5\": container with ID starting with ae8ba26088514c9618eeca849b3276201a5f19f149365178827b8d83c873a3c5 not found: ID does not exist"
Jul 09 18:58:49.226752 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.226730    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:58:49.231212 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:49.231192    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-56d5bcc4bd-xr42v"]
Jul 09 18:58:50.482189 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:58:50.482154    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" path="/var/lib/kubelet/pods/c96f11fa-7a81-41e1-9805-438460f2049f/volumes"
Jul 09 18:59:07.557122 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557087    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:07.557577 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557533    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerName="main"
Jul 09 18:59:07.557577 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557546    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerName="main"
Jul 09 18:59:07.557577 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557558    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="main"
Jul 09 18:59:07.557577 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557563    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="main"
Jul 09 18:59:07.557577 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557577    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="storage-initializer"
Jul 09 18:59:07.557786 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557583    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="storage-initializer"
Jul 09 18:59:07.557786 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96f11fa-7a81-41e1-9805-438460f2049f" containerName="main"
Jul 09 18:59:07.557786 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.557655    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17f7a5d-5e11-4935-9799-da75fd7e24c0" containerName="main"
Jul 09 18:59:07.561660 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.561643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.565292 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.565265    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xr9c2\""
Jul 09 18:59:07.565398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.565318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:59:07.565398 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.565318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 09 18:59:07.572520 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.572499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:07.607654 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.607601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmsp\" (UniqueName: \"kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.607835 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.607706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.607835 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.607765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.708385 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.708333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.708385 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.708401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.708646 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.708469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmsp\" (UniqueName: \"kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.708813 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.708789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.710872 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.710849    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.716514 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.716492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmsp\" (UniqueName: \"kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:07.871971 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:07.871884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:08.020271 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:08.020221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:08.020594 ip-10-0-141-82 kubenswrapper[2565]: W0709 18:59:08.020567    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba428cf_f7d2_4ec0_8d9a_c8c888706cdc.slice/crio-0b7ee902eb794b32849479deaaefeb5479ae3061013e45e1bb91ca43115609fd WatchSource:0}: Error finding container 0b7ee902eb794b32849479deaaefeb5479ae3061013e45e1bb91ca43115609fd: Status 404 returned error can't find the container with id 0b7ee902eb794b32849479deaaefeb5479ae3061013e45e1bb91ca43115609fd
Jul 09 18:59:08.299994 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:08.299959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" event={"ID":"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc","Type":"ContainerStarted","Data":"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"}
Jul 09 18:59:08.300216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:08.300012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" event={"ID":"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc","Type":"ContainerStarted","Data":"0b7ee902eb794b32849479deaaefeb5479ae3061013e45e1bb91ca43115609fd"}
Jul 09 18:59:08.300216 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:08.300135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:08.320138 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:08.320088    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" podStartSLOduration=1.320072817 podStartE2EDuration="1.320072817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:08.319412309 +0000 UTC m=+3370.406661612" watchObservedRunningTime="2026-07-09 18:59:08.320072817 +0000 UTC m=+3370.407322120"
Jul 09 18:59:39.307404 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:39.307372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:40.843843 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:40.843812    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:59:40.861452    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486" cmd=["/bin/sleep","15"]
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:40.861538    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" message="" err=<
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:59:40.861555    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerName="main" containerID="cri-o://4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"
Jul 09 18:59:40.865313 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:40.861634    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerName="main" containerID="cri-o://4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486" gracePeriod=60
Jul 09 18:59:41.125017 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.124988    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:41.209456 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.209425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mmsp\" (UniqueName: \"kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp\") pod \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") "
Jul 09 18:59:41.209630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.209472    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds\") pod \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") "
Jul 09 18:59:41.209630 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.209522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs\") pod \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\" (UID: \"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc\") "
Jul 09 18:59:41.209799 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.209775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" (UID: "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:41.211643 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.211611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" (UID: "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:59:41.211748 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.211670    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp" (OuterVolumeSpecName: "kube-api-access-2mmsp") pod "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" (UID: "4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc"). InnerVolumeSpecName "kube-api-access-2mmsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:59:41.310776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.310739    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmsp\" (UniqueName: \"kubernetes.io/projected/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-kube-api-access-2mmsp\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:59:41.310776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.310775    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:59:41.310776 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.310785    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 18:59:41.472724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.472687    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerID="4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486" exitCode=0
Jul 09 18:59:41.472724 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.472734    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"
Jul 09 18:59:41.472962 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.472775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" event={"ID":"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc","Type":"ContainerDied","Data":"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"}
Jul 09 18:59:41.472962 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.472824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj" event={"ID":"4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc","Type":"ContainerDied","Data":"0b7ee902eb794b32849479deaaefeb5479ae3061013e45e1bb91ca43115609fd"}
Jul 09 18:59:41.472962 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.472846    2565 scope.go:117] "RemoveContainer" containerID="4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"
Jul 09 18:59:41.482573 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.482556    2565 scope.go:117] "RemoveContainer" containerID="4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"
Jul 09 18:59:41.482817 ip-10-0-141-82 kubenswrapper[2565]: E0709 18:59:41.482801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486\": container with ID starting with 4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486 not found: ID does not exist" containerID="4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"
Jul 09 18:59:41.482897 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.482825    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486"} err="failed to get container status \"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486\": rpc error: code = NotFound desc = could not find container \"4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486\": container with ID starting with 4b785e065864aa8442c98c5c6ce790e208e8009755aa36d9cc77d127aac70486 not found: ID does not exist"
Jul 09 18:59:41.494983 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.494955    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:41.499117 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:41.499095    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87wk8bj"]
Jul 09 18:59:42.483226 ip-10-0-141-82 kubenswrapper[2565]: I0709 18:59:42.483185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" path="/var/lib/kubelet/pods/4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc/volumes"
Jul 09 19:00:07.926166 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.926122    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:00:07.926707 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.926688    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerName="main"
Jul 09 19:00:07.926785 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.926709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerName="main"
Jul 09 19:00:07.926875 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.926861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba428cf-f7d2-4ec0-8d9a-c8c888706cdc" containerName="main"
Jul 09 19:00:07.930272 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.930219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:07.934303 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.934277    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-vgjpl\""
Jul 09 19:00:07.934456 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.934313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 19:00:07.934456 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.934275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 19:00:07.944458 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.944428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:07.944607 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.944554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:07.944607 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.944600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnd4\" (UniqueName: \"kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:07.947827 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:07.947804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:00:08.045545 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.045511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.045743 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.045553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnd4\" (UniqueName: \"kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.045743 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.045617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.045903 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.045880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.048757 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.048728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.054099 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.054068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnd4\" (UniqueName: \"kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.240617 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.240568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.382195 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.382169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:00:08.383842 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:00:08.383810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f07a5cb_f9a6_4258_9430_4ce7f4d10bd2.slice/crio-a552c41624eb8bad17ce3bdd97caa541e33ea170b465e8fec7bfe8421c1270fb WatchSource:0}: Error finding container a552c41624eb8bad17ce3bdd97caa541e33ea170b465e8fec7bfe8421c1270fb: Status 404 returned error can't find the container with id a552c41624eb8bad17ce3bdd97caa541e33ea170b465e8fec7bfe8421c1270fb
Jul 09 19:00:08.615815 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.615705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" event={"ID":"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2","Type":"ContainerStarted","Data":"f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac"}
Jul 09 19:00:08.615815 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.615778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" event={"ID":"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2","Type":"ContainerStarted","Data":"a552c41624eb8bad17ce3bdd97caa541e33ea170b465e8fec7bfe8421c1270fb"}
Jul 09 19:00:08.615815 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.615816    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:08.638104 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:08.638048    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" podStartSLOduration=1.63803414 podStartE2EDuration="1.63803414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:08.635997202 +0000 UTC m=+3430.723246508" watchObservedRunningTime="2026-07-09 19:00:08.63803414 +0000 UTC m=+3430.725283444"
Jul 09 19:00:19.693300 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.693269    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:00:19.696916 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.696893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.700669 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.700645    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 19:00:19.700816 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.700690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-x6rhh\""
Jul 09 19:00:19.709603 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.709579    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:00:19.750391 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.750352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpsp\" (UniqueName: \"kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.750570 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.750423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.750570 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.750491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.851262 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.851197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.851475 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.851278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpsp\" (UniqueName: \"kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.851475 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.851344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.851652 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.851627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.854136 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.854111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:19.860492 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:19.860469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpsp\" (UniqueName: \"kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:20.007608 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.007512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:20.152289 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.152214    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:00:20.155134 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:00:20.155092    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939a4624_1a99_4594_8975_be628a867729.slice/crio-d12e3233d49a5dd3a3441e550bfd78d149d039f2185b214d59564f679fffb2bb WatchSource:0}: Error finding container d12e3233d49a5dd3a3441e550bfd78d149d039f2185b214d59564f679fffb2bb: Status 404 returned error can't find the container with id d12e3233d49a5dd3a3441e550bfd78d149d039f2185b214d59564f679fffb2bb
Jul 09 19:00:20.680461 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.680423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" event={"ID":"939a4624-1a99-4594-8975-be628a867729","Type":"ContainerStarted","Data":"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"}
Jul 09 19:00:20.680646 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.680481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" event={"ID":"939a4624-1a99-4594-8975-be628a867729","Type":"ContainerStarted","Data":"d12e3233d49a5dd3a3441e550bfd78d149d039f2185b214d59564f679fffb2bb"}
Jul 09 19:00:20.680646 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.680509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:00:20.698894 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:20.698842    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" podStartSLOduration=1.6988265089999999 podStartE2EDuration="1.698826509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:20.697488652 +0000 UTC m=+3442.784737967" watchObservedRunningTime="2026-07-09 19:00:20.698826509 +0000 UTC m=+3442.786075812"
Jul 09 19:00:39.623145 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:39.623116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:00:51.687827 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:00:51.687796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:01:12.257994 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.257947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:01:12.267465 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:01:12.267427    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:01:12.267465 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.267465 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.267465 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880" cmd=["/bin/sleep","15"]
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.267480    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" message="" err=<
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:01:12.267495    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" podUID="939a4624-1a99-4594-8975-be628a867729" containerName="main" containerID="cri-o://1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"
Jul 09 19:01:12.267738 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.267538    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" podUID="939a4624-1a99-4594-8975-be628a867729" containerName="main" containerID="cri-o://1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880" gracePeriod=60
Jul 09 19:01:12.547655 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.547626    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:01:12.633559 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.633524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lpsp\" (UniqueName: \"kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp\") pod \"939a4624-1a99-4594-8975-be628a867729\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") "
Jul 09 19:01:12.633984 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.633684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs\") pod \"939a4624-1a99-4594-8975-be628a867729\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") "
Jul 09 19:01:12.633984 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.633728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds\") pod \"939a4624-1a99-4594-8975-be628a867729\" (UID: \"939a4624-1a99-4594-8975-be628a867729\") "
Jul 09 19:01:12.634213 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.634185    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "939a4624-1a99-4594-8975-be628a867729" (UID: "939a4624-1a99-4594-8975-be628a867729"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:12.636192 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.636165    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "939a4624-1a99-4594-8975-be628a867729" (UID: "939a4624-1a99-4594-8975-be628a867729"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:01:12.636331 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.636304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp" (OuterVolumeSpecName: "kube-api-access-8lpsp") pod "939a4624-1a99-4594-8975-be628a867729" (UID: "939a4624-1a99-4594-8975-be628a867729"). InnerVolumeSpecName "kube-api-access-8lpsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:12.710961 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.710925    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:01:12.720479 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:01:12.720441    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:01:12.720479 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.720479 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.720479 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac" cmd=["/bin/sleep","15"]
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.720501    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" message="" err=<
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:01:12.720513    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerName="main" containerID="cri-o://f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac"
Jul 09 19:01:12.720765 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.720556    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerName="main" containerID="cri-o://f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac" gracePeriod=60
Jul 09 19:01:12.735011 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.734984    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/939a4624-1a99-4594-8975-be628a867729-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:12.735011 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.735008    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/939a4624-1a99-4594-8975-be628a867729-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:12.735224 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.735017    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpsp\" (UniqueName: \"kubernetes.io/projected/939a4624-1a99-4594-8975-be628a867729-kube-api-access-8lpsp\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:12.961876 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.961840    2565 generic.go:356] "Generic (PLEG): container finished" podID="939a4624-1a99-4594-8975-be628a867729" containerID="1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880" exitCode=0
Jul 09 19:01:12.962086 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.961920    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"
Jul 09 19:01:12.962086 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.961967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" event={"ID":"939a4624-1a99-4594-8975-be628a867729","Type":"ContainerDied","Data":"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"}
Jul 09 19:01:12.962086 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.962036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj" event={"ID":"939a4624-1a99-4594-8975-be628a867729","Type":"ContainerDied","Data":"d12e3233d49a5dd3a3441e550bfd78d149d039f2185b214d59564f679fffb2bb"}
Jul 09 19:01:12.962086 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.962065    2565 scope.go:117] "RemoveContainer" containerID="1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"
Jul 09 19:01:12.963976 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.963950    2565 generic.go:356] "Generic (PLEG): container finished" podID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerID="f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac" exitCode=0
Jul 09 19:01:12.964083 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.964034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" event={"ID":"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2","Type":"ContainerDied","Data":"f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac"}
Jul 09 19:01:12.972123 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.972105    2565 scope.go:117] "RemoveContainer" containerID="1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"
Jul 09 19:01:12.972489 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:01:12.972427    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880\": container with ID starting with 1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880 not found: ID does not exist" containerID="1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"
Jul 09 19:01:12.972489 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.972465    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880"} err="failed to get container status \"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880\": rpc error: code = NotFound desc = could not find container \"1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880\": container with ID starting with 1ab7d733bf5e0ac4de2a77520b4e2a3447a8db6dee38249dabc7155ae8d86880 not found: ID does not exist"
Jul 09 19:01:12.977177 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.977153    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:01:12.990601 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.990573    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:01:12.994032 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:12.994004    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85ddc2mj"]
Jul 09 19:01:13.037890 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.037860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds\") pod \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") "
Jul 09 19:01:13.038089 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.037988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs\") pod \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") "
Jul 09 19:01:13.038089 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.038055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwnd4\" (UniqueName: \"kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4\") pod \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\" (UID: \"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2\") "
Jul 09 19:01:13.038213 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.038181    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" (UID: "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:13.038344 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.038328    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:13.040202 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.040174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4" (OuterVolumeSpecName: "kube-api-access-kwnd4") pod "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" (UID: "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2"). InnerVolumeSpecName "kube-api-access-kwnd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:13.040202 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.040185    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" (UID: "1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:01:13.139584 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.139474    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwnd4\" (UniqueName: \"kubernetes.io/projected/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-kube-api-access-kwnd4\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:13.139584 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.139523    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:01:13.970579 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.970554    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"
Jul 09 19:01:13.971022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.970597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4" event={"ID":"1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2","Type":"ContainerDied","Data":"a552c41624eb8bad17ce3bdd97caa541e33ea170b465e8fec7bfe8421c1270fb"}
Jul 09 19:01:13.971022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.970661    2565 scope.go:117] "RemoveContainer" containerID="f12a4549c3fd69c811eb53a85d578521b6bfe726665adaa0ecdeaf2f8348dbac"
Jul 09 19:01:13.994020 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.993982    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:01:13.998084 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:13.998058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-m92n4"]
Jul 09 19:01:14.489177 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:14.489143    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" path="/var/lib/kubelet/pods/1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2/volumes"
Jul 09 19:01:14.489566 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:14.489551    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939a4624-1a99-4594-8975-be628a867729" path="/var/lib/kubelet/pods/939a4624-1a99-4594-8975-be628a867729/volumes"
Jul 09 19:01:35.162749 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.162667    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"]
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163043    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939a4624-1a99-4594-8975-be628a867729" containerName="main"
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163055    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="939a4624-1a99-4594-8975-be628a867729" containerName="main"
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163074    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerName="main"
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163079    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerName="main"
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f07a5cb-f9a6-4258-9430-4ce7f4d10bd2" containerName="main"
Jul 09 19:01:35.163253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.163147    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="939a4624-1a99-4594-8975-be628a867729" containerName="main"
Jul 09 19:01:35.167577 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.167560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.188138 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.188105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"]
Jul 09 19:01:35.339174 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.339130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-cert\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.339379 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.339192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl7m\" (UniqueName: \"kubernetes.io/projected/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-kube-api-access-krl7m\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.439951 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.439920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-cert\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.440128 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.439978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl7m\" (UniqueName: \"kubernetes.io/projected/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-kube-api-access-krl7m\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.442654 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.442624    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-cert\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.449595 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.449567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl7m\" (UniqueName: \"kubernetes.io/projected/73026d0b-9a52-4e79-bbc9-e56a5ef1ffee-kube-api-access-krl7m\") pod \"llmisvc-controller-manager-6b8b6f4bc5-n4wjm\" (UID: \"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee\") " pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.477618 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.477579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:35.626714 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.626687    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"]
Jul 09 19:01:35.628435 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:01:35.628388    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73026d0b_9a52_4e79_bbc9_e56a5ef1ffee.slice/crio-68215d59fd15b253711f9b79abd2e354ccb7715556cee1a19d69e6186ab90e7d WatchSource:0}: Error finding container 68215d59fd15b253711f9b79abd2e354ccb7715556cee1a19d69e6186ab90e7d: Status 404 returned error can't find the container with id 68215d59fd15b253711f9b79abd2e354ccb7715556cee1a19d69e6186ab90e7d
Jul 09 19:01:35.629854 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:35.629837    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:01:36.095905 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:36.095872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm" event={"ID":"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee","Type":"ContainerStarted","Data":"68215d59fd15b253711f9b79abd2e354ccb7715556cee1a19d69e6186ab90e7d"}
Jul 09 19:01:37.102822 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:37.102782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm" event={"ID":"73026d0b-9a52-4e79-bbc9-e56a5ef1ffee","Type":"ContainerStarted","Data":"a68d959583e4ed56e82f2b6303f8581ebd864ccc63303eadfca087d4183141d3"}
Jul 09 19:01:37.103287 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:37.102895    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:01:37.123745 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:01:37.123694    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm" podStartSLOduration=1.6994991590000001 podStartE2EDuration="2.123676977s" totalImagesPullingTime="424.177818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:37.121698121 +0000 UTC m=+3519.208947430" watchObservedRunningTime="2026-07-09 19:01:37.123676977 +0000 UTC m=+3519.210926280"
Jul 09 19:02:08.110816 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.110774    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b8b6f4bc5-n4wjm"
Jul 09 19:02:08.167030 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.166995    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 19:02:08.167286 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.167252    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" containerName="manager" containerID="cri-o://ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092" gracePeriod=30
Jul 09 19:02:08.425025 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.425000    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 19:02:08.516564 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.516533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9thcb\" (UniqueName: \"kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb\") pod \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") "
Jul 09 19:02:08.516731 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.516584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") pod \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\" (UID: \"73f547a2-9f2e-4b4e-822a-a22eb6f45de6\") "
Jul 09 19:02:08.518587 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.518557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert" (OuterVolumeSpecName: "cert") pod "73f547a2-9f2e-4b4e-822a-a22eb6f45de6" (UID: "73f547a2-9f2e-4b4e-822a-a22eb6f45de6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:02:08.518715 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.518601    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb" (OuterVolumeSpecName: "kube-api-access-9thcb") pod "73f547a2-9f2e-4b4e-822a-a22eb6f45de6" (UID: "73f547a2-9f2e-4b4e-822a-a22eb6f45de6"). InnerVolumeSpecName "kube-api-access-9thcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:02:08.617540 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.617498    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thcb\" (UniqueName: \"kubernetes.io/projected/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-kube-api-access-9thcb\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:02:08.617540 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:08.617532    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f547a2-9f2e-4b4e-822a-a22eb6f45de6-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:02:09.279971 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.279935    2565 generic.go:356] "Generic (PLEG): container finished" podID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" containerID="ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092" exitCode=0
Jul 09 19:02:09.280406 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.279997    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct"
Jul 09 19:02:09.280406 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.280020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" event={"ID":"73f547a2-9f2e-4b4e-822a-a22eb6f45de6","Type":"ContainerDied","Data":"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"}
Jul 09 19:02:09.280406 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.280082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fbfcc4968-m62ct" event={"ID":"73f547a2-9f2e-4b4e-822a-a22eb6f45de6","Type":"ContainerDied","Data":"aad683483efaeb98c961e3e851cf0a4511ecd310d4fd874b8b246aca1f5d9946"}
Jul 09 19:02:09.280406 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.280112    2565 scope.go:117] "RemoveContainer" containerID="ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"
Jul 09 19:02:09.290913 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.290895    2565 scope.go:117] "RemoveContainer" containerID="ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"
Jul 09 19:02:09.291156 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:02:09.291137    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092\": container with ID starting with ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092 not found: ID does not exist" containerID="ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"
Jul 09 19:02:09.291210 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.291167    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092"} err="failed to get container status \"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092\": rpc error: code = NotFound desc = could not find container \"ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092\": container with ID starting with ff08df59a606cd179f7fff359605192385632982d3d7789243d628036deb8092 not found: ID does not exist"
Jul 09 19:02:09.303264 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.303217    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 19:02:09.308578 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:09.308556    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-fbfcc4968-m62ct"]
Jul 09 19:02:10.482594 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:10.482559    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" path="/var/lib/kubelet/pods/73f547a2-9f2e-4b4e-822a-a22eb6f45de6/volumes"
Jul 09 19:02:46.356114 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.356076    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:02:46.356800 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.356525    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" containerName="manager"
Jul 09 19:02:46.356800 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.356543    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" containerName="manager"
Jul 09 19:02:46.356800 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.356709    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f547a2-9f2e-4b4e-822a-a22eb6f45de6" containerName="manager"
Jul 09 19:02:46.360058 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.360039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.362794 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.362772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-lxqg2\""
Jul 09 19:02:46.363774 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.363756    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 09 19:02:46.363839 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.363794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 19:02:46.370643 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.370620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:02:46.454875 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.454840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznrs\" (UniqueName: \"kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.455070 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.454941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.455070 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.454979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.556401 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.556363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznrs\" (UniqueName: \"kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.556401 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.556406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.556662 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.556425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.556791 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.556757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.559020 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.558990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.566304 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.566281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznrs\" (UniqueName: \"kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf228w\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.670532 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.670433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:46.828214 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:46.828051    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:02:46.831328 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:02:46.831297    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e39024_8237_49f3_8ac8_cc0a55f60fd3.slice/crio-f9cfd796f4c741eb3962f37b513d030e1b507eaa7f50ec55b6edfc203cdad1a8 WatchSource:0}: Error finding container f9cfd796f4c741eb3962f37b513d030e1b507eaa7f50ec55b6edfc203cdad1a8: Status 404 returned error can't find the container with id f9cfd796f4c741eb3962f37b513d030e1b507eaa7f50ec55b6edfc203cdad1a8
Jul 09 19:02:47.484641 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:47.484605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" event={"ID":"95e39024-8237-49f3-8ac8-cc0a55f60fd3","Type":"ContainerStarted","Data":"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"}
Jul 09 19:02:47.484641 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:47.484657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" event={"ID":"95e39024-8237-49f3-8ac8-cc0a55f60fd3","Type":"ContainerStarted","Data":"f9cfd796f4c741eb3962f37b513d030e1b507eaa7f50ec55b6edfc203cdad1a8"}
Jul 09 19:02:47.485101 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:47.484698    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:02:47.507623 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:02:47.507564    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" podStartSLOduration=1.507544698 podStartE2EDuration="1.507544698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:47.505755422 +0000 UTC m=+3589.593004728" watchObservedRunningTime="2026-07-09 19:02:47.507544698 +0000 UTC m=+3589.594794001"
Jul 09 19:03:10.040357 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.040325    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:03:10.045274 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.045247    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:03:10.049102 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.049077    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:03:10.052084 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.052063    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:03:10.053616 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.053597    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:03:10.059975 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:10.059951    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:03:12.342937 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.342901    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"]
Jul 09 19:03:12.346470 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.346445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.349285 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.349224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 09 19:03:12.358715 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.358690    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"]
Jul 09 19:03:12.385478 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.385673 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.385673 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.385673 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.385673 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac168a80-a025-47b8-bf9d-7e8862b3b452-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.385833 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.385674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnz2m\" (UniqueName: \"kubernetes.io/projected/ac168a80-a025-47b8-bf9d-7e8862b3b452-kube-api-access-cnz2m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486764 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486947 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486947 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac168a80-a025-47b8-bf9d-7e8862b3b452-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486947 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnz2m\" (UniqueName: \"kubernetes.io/projected/ac168a80-a025-47b8-bf9d-7e8862b3b452-kube-api-access-cnz2m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486947 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.486947 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.486904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.487224 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.487202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.487314 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.487295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.487380 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.487360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.489201 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.489172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac168a80-a025-47b8-bf9d-7e8862b3b452-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.489372 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.489273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac168a80-a025-47b8-bf9d-7e8862b3b452-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.495771 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.495747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnz2m\" (UniqueName: \"kubernetes.io/projected/ac168a80-a025-47b8-bf9d-7e8862b3b452-kube-api-access-cnz2m\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj\" (UID: \"ac168a80-a025-47b8-bf9d-7e8862b3b452\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.657676 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.657589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:12.804185 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:12.804155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"]
Jul 09 19:03:12.806072 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:03:12.806049    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac168a80_a025_47b8_bf9d_7e8862b3b452.slice/crio-480e857d5850b8163427f905559af5044ad2ed36ee6a2705ad6354040c74f0eb WatchSource:0}: Error finding container 480e857d5850b8163427f905559af5044ad2ed36ee6a2705ad6354040c74f0eb: Status 404 returned error can't find the container with id 480e857d5850b8163427f905559af5044ad2ed36ee6a2705ad6354040c74f0eb
Jul 09 19:03:13.623144 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:13.623099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj" event={"ID":"ac168a80-a025-47b8-bf9d-7e8862b3b452","Type":"ContainerStarted","Data":"a2c30a1a2a6a94250bbf07c037d19cd228bec7cbeb1ce9a900ab6ae7484c6aa8"}
Jul 09 19:03:13.623667 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:13.623164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj" event={"ID":"ac168a80-a025-47b8-bf9d-7e8862b3b452","Type":"ContainerStarted","Data":"480e857d5850b8163427f905559af5044ad2ed36ee6a2705ad6354040c74f0eb"}
Jul 09 19:03:13.646015 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:13.645936    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj" podStartSLOduration=1.645913027 podStartE2EDuration="1.645913027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:13.642318432 +0000 UTC m=+3615.729567737" watchObservedRunningTime="2026-07-09 19:03:13.645913027 +0000 UTC m=+3615.733162332"
Jul 09 19:03:13.657937 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:13.657902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:18.491147 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:18.491117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:03:22.658741 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:22.658700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:22.672475 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:22.672446    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:22.683908 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:22.683883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj"
Jul 09 19:03:34.097888 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.097839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:03:34.107960 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:03:34.107904    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:03:34.107960 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:34.107960 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:34.107960 ip-10-0-141-82 kubenswrapper[2565]:  > containerID="f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58" cmd=["/bin/sleep","15"]
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.107980    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" message="" err=<
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:  >
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:03:34.107994    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerName="main" containerID="cri-o://f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"
Jul 09 19:03:34.108332 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.108033    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerName="main" containerID="cri-o://f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58" gracePeriod=60
Jul 09 19:03:34.358777 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.358707    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:03:34.397597 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.397563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dznrs\" (UniqueName: \"kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs\") pod \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") "
Jul 09 19:03:34.397597 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.397595    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs\") pod \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") "
Jul 09 19:03:34.397835 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.397635    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds\") pod \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\" (UID: \"95e39024-8237-49f3-8ac8-cc0a55f60fd3\") "
Jul 09 19:03:34.398024 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.397994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "95e39024-8237-49f3-8ac8-cc0a55f60fd3" (UID: "95e39024-8237-49f3-8ac8-cc0a55f60fd3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:34.399725 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.399700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs" (OuterVolumeSpecName: "kube-api-access-dznrs") pod "95e39024-8237-49f3-8ac8-cc0a55f60fd3" (UID: "95e39024-8237-49f3-8ac8-cc0a55f60fd3"). InnerVolumeSpecName "kube-api-access-dznrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:03:34.399964 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.399949    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95e39024-8237-49f3-8ac8-cc0a55f60fd3" (UID: "95e39024-8237-49f3-8ac8-cc0a55f60fd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:03:34.498748 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.498719    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dznrs\" (UniqueName: \"kubernetes.io/projected/95e39024-8237-49f3-8ac8-cc0a55f60fd3-kube-api-access-dznrs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:03:34.498748 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.498745    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tls-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:03:34.498981 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.498761    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95e39024-8237-49f3-8ac8-cc0a55f60fd3-tokenizer-uds\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\""
Jul 09 19:03:34.739982 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.739944    2565 generic.go:356] "Generic (PLEG): container finished" podID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerID="f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58" exitCode=0
Jul 09 19:03:34.739982 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.739991    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"
Jul 09 19:03:34.740328 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.740027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" event={"ID":"95e39024-8237-49f3-8ac8-cc0a55f60fd3","Type":"ContainerDied","Data":"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"}
Jul 09 19:03:34.740328 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.740088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w" event={"ID":"95e39024-8237-49f3-8ac8-cc0a55f60fd3","Type":"ContainerDied","Data":"f9cfd796f4c741eb3962f37b513d030e1b507eaa7f50ec55b6edfc203cdad1a8"}
Jul 09 19:03:34.740328 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.740116    2565 scope.go:117] "RemoveContainer" containerID="f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"
Jul 09 19:03:34.751906 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.751885    2565 scope.go:117] "RemoveContainer" containerID="f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"
Jul 09 19:03:34.752180 ip-10-0-141-82 kubenswrapper[2565]: E0709 19:03:34.752158    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58\": container with ID starting with f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58 not found: ID does not exist" containerID="f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"
Jul 09 19:03:34.752228 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.752189    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58"} err="failed to get container status \"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58\": rpc error: code = NotFound desc = could not find container \"f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58\": container with ID starting with f13c6332a994ec98aded731849d5b6e2441eeb9da947f14f9a65c79ce9ed6d58 not found: ID does not exist"
Jul 09 19:03:34.763622 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.763595    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:03:34.765941 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:34.765919    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedf228w"]
Jul 09 19:03:36.481551 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:03:36.481516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" path="/var/lib/kubelet/pods/95e39024-8237-49f3-8ac8-cc0a55f60fd3/volumes"
Jul 09 19:08:10.074991 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.074961    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:08:10.079278 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.079255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:08:10.085671 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.085652    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:08:10.087551 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.087532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:08:10.090740 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.090721    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:08:10.097496 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:08:10.097476    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:13:10.111776 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.111674    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:13:10.116382 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.116355    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:13:10.123635 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.123616    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:13:10.123865 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.123846    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:13:10.128221 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.128204    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:13:10.134712 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:13:10.134696    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:18:10.151481 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.151364    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:18:10.155972 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.155950    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:18:10.163120 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.163098    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:18:10.163847 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.163830    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:18:10.168146 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.168127    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:18:10.174791 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:18:10.174775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:22:10.628382 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:10.628346    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:30.405094 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:30.405013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:30.479521 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:30.479493    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:30.479781 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:30.479717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:30.555937 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:30.555908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:32.075781 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:32.075745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:32.117256 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:32.117203    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:32.117428 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:32.117328    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:32.160682 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:32.160655    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:33.700878 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:33.700847    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:33.739695 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:33.739669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:33.739871 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:33.739768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:33.780982 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:33.780954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:35.307803 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:35.307780    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:35.349867 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:35.349837    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:35.350045 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:35.350028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:35.394858 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:35.394830    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:36.919070 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:36.919042    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:36.956064 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:36.956031    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:36.956247 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:36.956199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:37.001135 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:37.001104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:38.554729 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:38.554705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:38.606817 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:38.606784    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:38.606994 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:38.606831    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:38.651387 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:38.651356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:40.162841 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:40.162812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:40.201048 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:40.201020    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:40.201205 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:40.201120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:40.250938 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:40.250906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:41.893615 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:41.893582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:41.944144 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:41.944111    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:41.944144 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:41.944130    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:41.996199 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:41.996165    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:43.637294 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:43.637263    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:43.679168 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:43.679146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:43.679356 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:43.679187    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:43.728630 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:43.728603    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:45.243816 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:45.243786    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:45.303132 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:45.303090    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:45.303132 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:45.303109    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:45.347800 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:45.347773    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:46.896817 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:46.896757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:46.934825 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:46.934793    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:46.935035 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:46.934891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:46.975163 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:46.975119    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:48.572101 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:48.572069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:48.615822 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:48.615796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:48.616002 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:48.615954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:48.674518 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:48.674493    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:50.283805 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:50.283778    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:50.358447 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:50.358419    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:50.358649 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:50.358567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:50.417317 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:50.417276    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:51.937655 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:51.937630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:51.974008 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:51.973980    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:51.974215 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:51.974088    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:52.020387 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:52.020363    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:53.575389 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:53.575341    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:53.615678 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:53.615651    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:53.615868 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:53.615856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:53.662224 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:53.662199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:55.228205 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:55.228156    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-24fsq_45d95cc2-d0b6-40a5-88e2-e4dcf0f00eb5/storage-initializer/0.log"
Jul 09 19:22:55.294985 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:55.294958    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/0.log"
Jul 09 19:22:55.299054 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:55.299020    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-m4ddf_da5f8378-56ff-4c15-b881-d6d7b5ffb8b9/istio-proxy/1.log"
Jul 09 19:22:55.389494 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:55.389466    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-kz6fj_ac168a80-a025-47b8-bf9d-7e8862b3b452/main/0.log"
Jul 09 19:22:57.197253 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:57.197190    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:22:57.211970 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:57.211939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:22:57.229247 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:57.229212    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:22:57.244107 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:57.244081    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:22:58.460830 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:58.460801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:22:58.480981 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:58.480948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:22:58.498332 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:58.498307    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:22:58.514531 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:58.514506    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:22:59.742952 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:59.742913    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:22:59.762297 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:59.762263    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:22:59.779461 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:59.779425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:22:59.792413 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:22:59.792384    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:23:00.966818 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:00.966789    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:23:00.989295 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:00.989260    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:23:01.010707 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:01.010675    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:23:01.023867 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:01.023842    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:23:02.364212 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:02.364179    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:03.715023 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:03.714995    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:05.062856 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:05.062814    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:06.386263 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:06.386220    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:07.730022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:07.729974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:09.080396 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:09.080362    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:10.187132 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.187019    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:23:10.193172 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.191481    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:23:10.198247 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.198210    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:23:10.200389 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.200372    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:23:10.204303 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.204287    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:23:10.210564 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.210547    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqxhn_dd9e7bc5-49c5-4535-b31c-b9f2b594e74f/ovn-acl-logging/0.log"
Jul 09 19:23:10.414758 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:10.414722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:11.836382 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:11.836356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:13.261362 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:13.261334    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:14.607972 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:14.607948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:15.926518 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:15.926488    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:18.568197 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.568166    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vv9mp/must-gather-t6bfs"]
Jul 09 19:23:18.568598 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.568584    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerName="main"
Jul 09 19:23:18.568647 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.568599    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerName="main"
Jul 09 19:23:18.568683 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.568664    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e39024-8237-49f3-8ac8-cc0a55f60fd3" containerName="main"
Jul 09 19:23:18.570800 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.570784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.576724 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.576697    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vv9mp\"/\"kube-root-ca.crt\""
Jul 09 19:23:18.576892 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.576733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vv9mp\"/\"default-dockercfg-rrxh9\""
Jul 09 19:23:18.577644 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.577624    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vv9mp\"/\"openshift-service-ca.crt\""
Jul 09 19:23:18.587849 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.587827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/must-gather-t6bfs"]
Jul 09 19:23:18.713348 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.713305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdpj7\" (UniqueName: \"kubernetes.io/projected/f62a31cb-63aa-464a-b731-11306c5d80f8-kube-api-access-tdpj7\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.713542 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.713403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62a31cb-63aa-464a-b731-11306c5d80f8-must-gather-output\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.814851 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.814813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdpj7\" (UniqueName: \"kubernetes.io/projected/f62a31cb-63aa-464a-b731-11306c5d80f8-kube-api-access-tdpj7\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.815061 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.814868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62a31cb-63aa-464a-b731-11306c5d80f8-must-gather-output\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.815198 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.815179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f62a31cb-63aa-464a-b731-11306c5d80f8-must-gather-output\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.824409 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.824349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdpj7\" (UniqueName: \"kubernetes.io/projected/f62a31cb-63aa-464a-b731-11306c5d80f8-kube-api-access-tdpj7\") pod \"must-gather-t6bfs\" (UID: \"f62a31cb-63aa-464a-b731-11306c5d80f8\") " pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:18.891527 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:18.891491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/must-gather-t6bfs"
Jul 09 19:23:19.028388 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:19.028359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/must-gather-t6bfs"]
Jul 09 19:23:19.030033 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:23:19.030009    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf62a31cb_63aa_464a_b731_11306c5d80f8.slice/crio-539e32fc7b629854156a3d2f8920c1775951f1f1d2e6a430f30e81def51deaf0 WatchSource:0}: Error finding container 539e32fc7b629854156a3d2f8920c1775951f1f1d2e6a430f30e81def51deaf0: Status 404 returned error can't find the container with id 539e32fc7b629854156a3d2f8920c1775951f1f1d2e6a430f30e81def51deaf0
Jul 09 19:23:19.031893 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:19.031870    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:23:20.029077 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:20.028982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/must-gather-t6bfs" event={"ID":"f62a31cb-63aa-464a-b731-11306c5d80f8","Type":"ContainerStarted","Data":"cd100a62a4ec693a341509a9ca877ae6f426b325a580daec063bac191f434452"}
Jul 09 19:23:20.029077 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:20.029038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/must-gather-t6bfs" event={"ID":"f62a31cb-63aa-464a-b731-11306c5d80f8","Type":"ContainerStarted","Data":"539e32fc7b629854156a3d2f8920c1775951f1f1d2e6a430f30e81def51deaf0"}
Jul 09 19:23:21.042498 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:21.042452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/must-gather-t6bfs" event={"ID":"f62a31cb-63aa-464a-b731-11306c5d80f8","Type":"ContainerStarted","Data":"7f8cbf8a760944dd00dc7819c284a1f9759faf7b71b3723d8c8fad5622043f08"}
Jul 09 19:23:21.062706 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:21.062634    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vv9mp/must-gather-t6bfs" podStartSLOduration=2.199950728 podStartE2EDuration="3.062612757s" totalImagesPullingTime="862.662029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:21.06061387 +0000 UTC m=+4823.147863176" watchObservedRunningTime="2026-07-09 19:23:21.062612757 +0000 UTC m=+4823.149862062"
Jul 09 19:23:21.582680 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:21.582648    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kmbmg_24b38b34-371d-43da-9ed4-6da5eb6ee973/global-pull-secret-syncer/0.log"
Jul 09 19:23:21.722015 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:21.721959    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-87jjw_3022f01d-3e8d-4f25-964b-f0cfbdd07da6/konnectivity-agent/0.log"
Jul 09 19:23:21.826688 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:21.826653    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-82.ec2.internal_52024e858ee987f467ceba75222ec174/haproxy/0.log"
Jul 09 19:23:26.847882 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:26.847837    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-8kx2v_caef0821-ceef-4284-b1a9-b5ea5ec283d0/manager/0.log"
Jul 09 19:23:28.453031 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:28.452996    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/1.log"
Jul 09 19:23:28.580518 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:28.579980    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-v7s7l_5073d86a-582f-4f04-ad5a-130ea337c439/cluster-monitoring-operator/0.log"
Jul 09 19:23:28.995315 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:28.995260    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtb6d_5c171010-2af7-4c10-8d31-f785c24d7900/node-exporter/0.log"
Jul 09 19:23:29.018856 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:29.018829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtb6d_5c171010-2af7-4c10-8d31-f785c24d7900/kube-rbac-proxy/0.log"
Jul 09 19:23:29.049026 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:29.049001    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtb6d_5c171010-2af7-4c10-8d31-f785c24d7900/init-textfile/0.log"
Jul 09 19:23:30.789757 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:30.789726    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-zml89_e9a8c762-322e-4594-b759-311e8ed6b8ee/networking-console-plugin/0.log"
Jul 09 19:23:31.301537 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.301510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/2.log"
Jul 09 19:23:31.307427 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.307402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-89h9v_ab136634-580b-4ce5-80fa-6c52e0e7e747/console-operator/3.log"
Jul 09 19:23:31.346847 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.346804    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"]
Jul 09 19:23:31.356367 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.356331    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.368753 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.368722    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"]
Jul 09 19:23:31.454814 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.454780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-proc\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.455022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.454825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wf7m\" (UniqueName: \"kubernetes.io/projected/e9083fcc-2341-4a22-a542-9361e15bccc2-kube-api-access-2wf7m\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.455022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.454860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-lib-modules\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.455022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.454924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-podres\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.455022 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.455007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-sys\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555612 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-proc\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555612 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wf7m\" (UniqueName: \"kubernetes.io/projected/e9083fcc-2341-4a22-a542-9361e15bccc2-kube-api-access-2wf7m\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-lib-modules\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-proc\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-podres\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-podres\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-lib-modules\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.555864 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-sys\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.556203 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.555976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9083fcc-2341-4a22-a542-9361e15bccc2-sys\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.565016 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.564993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wf7m\" (UniqueName: \"kubernetes.io/projected/e9083fcc-2341-4a22-a542-9361e15bccc2-kube-api-access-2wf7m\") pod \"perf-node-gather-daemonset-c49qm\" (UID: \"e9083fcc-2341-4a22-a542-9361e15bccc2\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.671463 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.671427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:31.831436 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.830291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7599f89d75-dfw7d_18ab3736-5d7a-4d25-87cb-b5f283bf4a9b/console/0.log"
Jul 09 19:23:31.846599 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:31.846556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"]
Jul 09 19:23:31.850069 ip-10-0-141-82 kubenswrapper[2565]: W0709 19:23:31.850031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9083fcc_2341_4a22_a542_9361e15bccc2.slice/crio-5fca5613f088b7f1ffb22f0e0ba22132b4feb159b3784c8dcc20d6635d81015f WatchSource:0}: Error finding container 5fca5613f088b7f1ffb22f0e0ba22132b4feb159b3784c8dcc20d6635d81015f: Status 404 returned error can't find the container with id 5fca5613f088b7f1ffb22f0e0ba22132b4feb159b3784c8dcc20d6635d81015f
Jul 09 19:23:32.137578 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:32.137489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm" event={"ID":"e9083fcc-2341-4a22-a542-9361e15bccc2","Type":"ContainerStarted","Data":"60cb5052c90a4113fcf691ab41ed05f70c199d0c4b542ddad747d0bf9fbb9986"}
Jul 09 19:23:32.137578 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:32.137554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm" event={"ID":"e9083fcc-2341-4a22-a542-9361e15bccc2","Type":"ContainerStarted","Data":"5fca5613f088b7f1ffb22f0e0ba22132b4feb159b3784c8dcc20d6635d81015f"}
Jul 09 19:23:32.138481 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:32.138453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:32.159203 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:32.159138    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm" podStartSLOduration=1.159115026 podStartE2EDuration="1.159115026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:32.15742967 +0000 UTC m=+4834.244678975" watchObservedRunningTime="2026-07-09 19:23:32.159115026 +0000 UTC m=+4834.246364331"
Jul 09 19:23:32.436315 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:32.436227    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-pqf7p_d0f5cfa2-5396-418f-b708-79e38e746ac4/volume-data-source-validator/0.log"
Jul 09 19:23:33.557918 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:33.557876    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt2s9_1d432f1b-b963-4d1a-bec9-052434b9454d/dns/0.log"
Jul 09 19:23:33.581715 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:33.581686    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt2s9_1d432f1b-b963-4d1a-bec9-052434b9454d/kube-rbac-proxy/0.log"
Jul 09 19:23:33.608860 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:33.608832    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82jsv_6430784f-f90b-4d10-9461-d80c0d848b40/dns-node-resolver/0.log"
Jul 09 19:23:34.308073 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:34.308044    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdz76_44981ba1-e2ac-4bca-83c9-6988a5237880/node-ca/0.log"
Jul 09 19:23:35.652129 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:35.652096    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:23:35.680471 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:35.680439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:23:35.705691 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:35.705659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:23:35.730111 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:35.730084    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:23:36.300868 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.300838    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgpt5_0af21ee9-3d83-4aa9-a541-ddb07176e777/serve-healthcheck-canary/0.log"
Jul 09 19:23:36.768794 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.768763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-x6k6c_44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3/insights-operator/0.log"
Jul 09 19:23:36.771837 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.771812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-x6k6c_44c78c0b-6eaf-4d46-8ae4-3e3544b25fa3/insights-operator/1.log"
Jul 09 19:23:36.943159 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.943133    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4475_9d071fa4-9ca5-46d3-85cd-c4af35330ddf/kube-rbac-proxy/0.log"
Jul 09 19:23:36.967569 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.967540    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4475_9d071fa4-9ca5-46d3-85cd-c4af35330ddf/exporter/0.log"
Jul 09 19:23:36.990476 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:36.990452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4475_9d071fa4-9ca5-46d3-85cd-c4af35330ddf/extractor/0.log"
Jul 09 19:23:39.163773 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:39.163739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-c49qm"
Jul 09 19:23:40.940226 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:40.940195    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-748bdf6488-jqrkk_6fe0bac6-5817-4f9b-ac0a-af163f0c81d7/manager/0.log"
Jul 09 19:23:40.986414 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:40.986370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b8b6f4bc5-n4wjm_73026d0b-9a52-4e79-bbc9-e56a5ef1ffee/manager/0.log"
Jul 09 19:23:41.013264 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:41.013222    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-jpp7h_c422f8f1-a005-443e-bda7-4d0e12b5158c/server/0.log"
Jul 09 19:23:41.671520 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:41.671488    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-chmc9_c3283ae9-c571-46cd-b755-6649389b912d/manager/0.log"
Jul 09 19:23:43.180305 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:43.180198    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:23:43.207052 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:43.207023    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:23:43.231698 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:43.231667    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:23:43.245985 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:43.245936    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:23:44.605263 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:44.605089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4wpb9_4c422460-a5ad-4b42-95a0-0fb105c1a60e/discovery/0.log"
Jul 09 19:23:44.633522 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:44.633481    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/0.log"
Jul 09 19:23:44.661618 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:44.661570    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-wqvjb_ff85a442-d24b-4a72-a758-81e518cfd1fd/istio-proxy/1.log"
Jul 09 19:23:44.680063 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:44.680036    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fd9c96588-sbl55_bd3fdf67-d3ba-4cb1-90ef-0ea9498ae4e0/router/0.log"
Jul 09 19:23:46.901787 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:46.901763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-qx52v_3ae4b84a-688a-42ff-9feb-3463e8cc35d5/migrator/0.log"
Jul 09 19:23:46.924302 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:46.924269    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-qx52v_3ae4b84a-688a-42ff-9feb-3463e8cc35d5/graceful-termination/0.log"
Jul 09 19:23:47.353796 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:47.353764    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-f5qgx_b5b51860-1dbb-4ff1-a7cb-0273f891f537/kube-storage-version-migrator-operator/1.log"
Jul 09 19:23:47.354760 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:47.354726    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-f5qgx_b5b51860-1dbb-4ff1-a7cb-0273f891f537/kube-storage-version-migrator-operator/0.log"
Jul 09 19:23:48.396313 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.396274    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/kube-multus-additional-cni-plugins/0.log"
Jul 09 19:23:48.421728 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.421697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/egress-router-binary-copy/0.log"
Jul 09 19:23:48.446647 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.446623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/cni-plugins/0.log"
Jul 09 19:23:48.475927 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.475891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/bond-cni-plugin/0.log"
Jul 09 19:23:48.502046 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.502018    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/routeoverride-cni/0.log"
Jul 09 19:23:48.526246 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.526199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/whereabouts-cni-bincopy/0.log"
Jul 09 19:23:48.558189 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:48.558158    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbckn_4df6f073-d9d4-4339-8fa3-8605f157a4e2/whereabouts-cni/0.log"
Jul 09 19:23:49.000974 ip-10-0-141-82 kubenswrapper[2565]: I0709 19:23:49.000942    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zwftt_6b0a56a1-54a9-4456-837a-a7ae772a9716/kube-multus/0.log"