Jul 15 10:45:34.048333 ip-10-0-137-92 systemd[1]: Starting Kubernetes Kubelet...
Jul 15 10:45:34.437585 ip-10-0-137-92 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:34.437585 ip-10-0-137-92 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 15 10:45:34.437585 ip-10-0-137-92 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:34.437585 ip-10-0-137-92 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 15 10:45:34.437585 ip-10-0-137-92 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:34.438594 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.438510    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 15 10:45:34.443959 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443943    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.443959 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443960    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443964    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443967    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443970    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443973    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443979    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443989    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443993    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443996    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.443998    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444001    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444004    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444006    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444009    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444012    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444014    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444016    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444019    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444021    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.444024 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444024    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444027    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444029    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444032    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444035    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444050    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444053    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444056    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444059    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444061    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444064    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444066    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444069    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444071    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444074    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444076    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444079    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444081    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444084    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444087    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.444486 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444089    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444091    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444094    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444096    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444099    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444102    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444104    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444107    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444109    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444112    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444115    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444118    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444120    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444122    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444126    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444129    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444131    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444134    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444137    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444139    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.444960 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444141    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444144    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444147    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444149    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444151    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444154    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444156    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444159    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444163    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444167    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444170    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444173    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444176    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444178    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444182    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444184    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444186    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444189    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444191    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444193    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.445455 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444196    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444199    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444201    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444203    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444206    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444210    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444214    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444217    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444220    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444222    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444225    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444228    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444230    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444233    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444235    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444238    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444241    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444243    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444246    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.445935 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444917    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444926    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444929    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444932    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444935    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444938    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444940    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444943    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444946    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444948    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444950    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444953    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444955    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444958    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444960    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444962    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444965    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444968    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444970    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.446413 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444975    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444979    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444982    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444984    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444987    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444990    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444993    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444995    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.444998    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445000    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445003    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445005    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445008    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445010    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445014    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445016    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445018    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445021    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445023    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445027    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.446892 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445029    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445032    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445034    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445036    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445054    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445057    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445061    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445064    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445068    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445071    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445074    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445077    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445080    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445083    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445085    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445088    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445091    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445094    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445097    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445099    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.447408 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445102    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445104    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445107    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445110    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445112    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445114    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445118    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445121    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445123    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445126    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445129    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445132    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445135    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445137    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445140    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445142    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445145    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445147    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445151    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.447896 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445154    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445157    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445159    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445161    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445164    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445168    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445170    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445173    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445175    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445178    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445180    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445183    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445185    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445188    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445191    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445193    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445195    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445198    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445200    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445204    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.448417 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445207    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445566    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445572    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445574    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445577    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445584    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445587    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445590    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445593    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445595    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445598    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445600    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445603    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445606    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445609    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445611    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445614    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445619    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445621    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445624    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.448932 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445627    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445629    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445633    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445637    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445640    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445643    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445647    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445650    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445653    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445656    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445659    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445663    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445665    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445668    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445670    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445673    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445675    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445677    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445680    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.449445 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445683    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445686    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445688    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445691    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445693    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445696    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445698    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445701    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445703    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445705    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445709    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445711    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445714    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445716    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445718    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445721    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445723    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445726    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445728    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.449905 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445731    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445733    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445735    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445738    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445740    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445745    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445747    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445750    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445752    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445754    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445757    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445759    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445761    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445764    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445767    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445769    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445771    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445774    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445776    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445779    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.450407 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445781    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445784    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445786    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445789    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445792    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445794    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445797    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445799    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445801    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445804    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445806    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445808    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445811    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445813    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445815    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445818    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445820    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445824    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445826    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445828    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.450889 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445831    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.445833    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445915    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445925    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445934    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445941    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445948    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445952    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445956    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445961    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445964    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445967    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445970    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445973    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445976    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445981    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445984    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445987    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445989    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.445993    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446000    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446003    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446006    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446009    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 15 10:45:34.451463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446012    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446016    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446019    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446022    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446025    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446029    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446032    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446035    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446051    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446054    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446058    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446061    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446064    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446067    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446069    2565 flags.go:64] FLAG: --enable-server="true"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446073    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446077    2565 flags.go:64] FLAG: --event-burst="100"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446080    2565 flags.go:64] FLAG: --event-qps="50"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446083    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446086    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446089    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446093    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446095    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446100    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446103    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 15 10:45:34.452063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446106    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446109    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446112    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446114    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446117    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446120    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446123    2565 flags.go:64] FLAG: --feature-gates=""
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446127    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446129    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446133    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446136    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446139    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446143    2565 flags.go:64] FLAG: --help="false"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446146    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446149    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446152    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446154    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446158    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446161    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446164    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446167    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446169    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446172    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 15 10:45:34.452647 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446176    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446179    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446181    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446184    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446187    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446190    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446193    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446197    2565 flags.go:64] FLAG: --lock-file=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446199    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446202    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446205    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446210    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446213    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446216    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446219    2565 flags.go:64] FLAG: --logging-format="text"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446222    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446225    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446228    2565 flags.go:64] FLAG: --manifest-url=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446231    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446235    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446238    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446243    2565 flags.go:64] FLAG: --max-pods="110"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446246    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446249    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446252    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 15 10:45:34.453246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446255    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446258    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446261    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446264    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446270    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446273    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446276    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446279    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446283    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446288    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446291    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446294    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446297    2565 flags.go:64] FLAG: --port="10250"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446299    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446302    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c4697f8fa4705065"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446306    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446309    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446312    2565 flags.go:64] FLAG: --register-node="true"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446315    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446318    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446321    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446324    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446326    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446330    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446333    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 15 10:45:34.453880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446336    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446338    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446341    2565 flags.go:64] FLAG: --runonce="false"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446346    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446348    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446351    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446354    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446357    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446360    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446363    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446367    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446369    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446372    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446375    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446378    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446382    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446385    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446388    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446393    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446395    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446398    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446403    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446407    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446409    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446412    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 15 10:45:34.454504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446415    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446418    2565 flags.go:64] FLAG: --v="2"
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446422    2565 flags.go:64] FLAG: --version="false"
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446426    2565 flags.go:64] FLAG: --vmodule=""
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446430    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446434    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446565    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446569    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446573    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446576    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446580    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446583    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446586    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446589    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446591    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446594    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446597    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446599    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446602    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446604    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446607    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.455152 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446609    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446612    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446614    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446617    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446620    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446622    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446625    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446627    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446631    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446634    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446636    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446639    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446641    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446644    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446647    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446651    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446653    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446656    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446660    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.455671 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446663    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446665    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446670    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446672    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446675    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446679    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446682    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446685    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446688    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446690    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446693    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446695    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446698    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446700    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446703    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446705    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446708    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446711    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446713    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.456142 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446716    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446718    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446722    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446725    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446727    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446730    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446732    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446734    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446737    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446739    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446742    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446744    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446747    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446749    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446751    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446755    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446758    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446760    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446762    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446765    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.456597 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446767    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446770    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446772    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446774    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446777    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446779    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446782    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446784    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446786    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446789    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446791    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446794    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446797    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446799    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446803    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446805    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446808    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446810    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446813    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446815    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.457177 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446818    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446820    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446823    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446825    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446828    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.446830    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.446835    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.455168    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.455183    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.455803    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.455810    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455867    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455872    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455876    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.457650 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455879    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455883    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455886    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455889    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455891    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455894    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455896    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455899    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455901    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455904    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455907    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455910    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455912    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455917    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455919    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455921    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455924    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455927    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455929    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.457993 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455932    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455934    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455937    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455939    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455942    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455945    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455948    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455951    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455954    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455956    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455959    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455961    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455964    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455966    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455969    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455971    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455974    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455976    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455979    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455981    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.458524 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455983    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455986    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455989    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455991    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455994    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455996    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.455998    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456001    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456005    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456009    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456012    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456014    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456017    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456020    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456023    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456025    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456028    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456031    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456034    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456036    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.458998 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456052    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456055    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456058    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456060    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456063    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456065    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456069    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456073    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456076    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456078    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456081    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456084    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456087    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456090    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456092    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456095    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456098    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456101    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456103    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456106    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.459514 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456108    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456110    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456113    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456115    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456117    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456120    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456122    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456125    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456127    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456130    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456133    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456137    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456140    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456143    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456146    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.459989 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456148    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.456154    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456263    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456268    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456270    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456273    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456276    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456278    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456280    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456283    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456285    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456288    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456291    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456295    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456297    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:34.460373 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456299    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456302    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456304    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456306    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456309    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456311    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456314    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456316    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456318    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456321    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456323    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456326    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456329    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456331    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456335    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456338    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456341    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456344    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456346    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456348    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:34.460734 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456351    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456353    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456356    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456358    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456360    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456363    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456365    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456367    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456370    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456372    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456374    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456380    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456384    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456387    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456389    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456392    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456394    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456397    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456400    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:34.461325 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456402    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456405    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456408    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456410    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456413    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456416    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456418    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456420    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456423    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456425    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456428    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456430    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456433    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456435    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456437    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456440    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456442    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456445    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456447    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456450    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:34.461772 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456452    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456455    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456457    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456460    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456463    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456466    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456469    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456471    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456474    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456476    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456478    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456481    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456483    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456485    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456488    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456491    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456493    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456496    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456498    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:34.462249 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456501    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456503    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456506    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456508    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456510    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456513    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456515    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:34.456518    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.456522    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.456620    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.458526    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.459742    2565 server.go:1019] "Starting client certificate rotation"
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.459847    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:45:34.462702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.460733    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:45:34.463024 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.462997    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:45:34.468766 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.468741    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:45:34.482656 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.482635    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 15 10:45:34.484449 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.484433    2565 log.go:25] "Validated CRI v1 image API"
Jul 15 10:45:34.485200 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.485185    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 15 10:45:34.487558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.487538    2565 fs.go:135] Filesystem UUIDs: map[035a7a9b-7cbc-493b-afb7-30fc102deda1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c65606d0-b859-438a-959b-668c803123a6:/dev/nvme0n1p3]
Jul 15 10:45:34.487637 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.487556    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 15 10:45:34.491773 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.491663    2565 manager.go:216] Machine: {Timestamp:2026-07-15 10:45:34.491171101 +0000 UTC m=+0.325915563 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099164 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d127c7bcdda9a750144f85de10631 SystemUUID:ec2d127c-7bcd-da9a-7501-44f85de10631 BootID:e80a5184-c24f-4e48-be81-bf3b19fcf74c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0c:1f:a0:9a:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0c:1f:a0:9a:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:e6:b3:54:93:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 15 10:45:34.491773 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.491768    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 15 10:45:34.491921 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.491844    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 15 10:45:34.495429 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.495403    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 15 10:45:34.495566 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.495429    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 15 10:45:34.496376 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.496366    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 15 10:45:34.496425 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.496379    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 15 10:45:34.496425 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.496386    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:45:34.497815 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.497805    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:45:34.498174 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.498157    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:45:34.499165 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.499151    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 15 10:45:34.502098 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502084    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:45:34.502236 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502225    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 15 10:45:34.502303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502265    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 15 10:45:34.502303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502288    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 15 10:45:34.502390 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502311    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 15 10:45:34.502390 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502328    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 15 10:45:34.502390 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502341    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 15 10:45:34.502725 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502711    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:45:34.502804 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.502737    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:45:34.504846 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.504826    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 15 10:45:34.506595 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506579    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 15 10:45:34.506663 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506627    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 15 10:45:34.506834 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506822    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 15 10:45:34.506867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506846    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 15 10:45:34.506867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506858    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 15 10:45:34.506924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506868    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 15 10:45:34.506924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506885    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 15 10:45:34.506924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.506896    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 15 10:45:34.507583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507573    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 15 10:45:34.507583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507584    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 15 10:45:34.507646 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507591    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 15 10:45:34.507646 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507597    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 15 10:45:34.507646 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507610    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 15 10:45:34.507646 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.507618    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 15 10:45:34.508464 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.508453    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 15 10:45:34.508496 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.508466    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 15 10:45:34.511763 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.511749    2565 server.go:1268] "Started kubelet"
Jul 15 10:45:34.511868 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.511840    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 15 10:45:34.511927 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.511847    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 15 10:45:34.511975 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.511933    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 15 10:45:34.515556 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.515522    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 15 10:45:34.515878 ip-10-0-137-92 systemd[1]: Started Kubernetes Kubelet.
Jul 15 10:45:34.516533 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.516403    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 15 10:45:34.516820 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.516724    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 15 10:45:34.517025 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.516996    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 15 10:45:34.518287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.518270    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 15 10:45:34.518819 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.518795    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 15 10:45:34.522107 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.521206    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-92.ec2.internal.18c270b731f6c752  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-92.ec2.internal,UID:ip-10-0-137-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-92.ec2.internal,},FirstTimestamp:2026-07-15 10:45:34.511720274 +0000 UTC m=+0.346464775,LastTimestamp:2026-07-15 10:45:34.511720274 +0000 UTC m=+0.346464775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-92.ec2.internal,}"
Jul 15 10:45:34.523975 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.523958    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 15 10:45:34.524033 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.523977    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 15 10:45:34.524394 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.524361    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:34.524580 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524561    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 15 10:45:34.524699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524563    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 15 10:45:34.524755 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524737    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 15 10:45:34.524755 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524700    2565 factory.go:55] Registering systemd factory
Jul 15 10:45:34.524855 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524772    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 15 10:45:34.525009 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.524992    2565 factory.go:153] Registering CRI-O factory
Jul 15 10:45:34.525009 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525008    2565 factory.go:223] Registration of the crio container factory successfully
Jul 15 10:45:34.525161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525053    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 15 10:45:34.525161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525064    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 15 10:45:34.525161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525102    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 15 10:45:34.525161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525126    2565 factory.go:103] Registering Raw factory
Jul 15 10:45:34.525161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525142    2565 manager.go:1196] Started watching for new ooms in manager
Jul 15 10:45:34.525623 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.525607    2565 manager.go:349] Starting recovery of all containers
Jul 15 10:45:34.526582 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.526508    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgw68"
Jul 15 10:45:34.532547 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.532513    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 15 10:45:34.533988 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.533965    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgw68"
Jul 15 10:45:34.535156 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.535129    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 15 10:45:34.535250 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.535216    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 15 10:45:34.535457 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.535439    2565 manager.go:354] Recovery completed
Jul 15 10:45:34.540331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.540224    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.542490 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.542476    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.542542 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.542507    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.542542 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.542521    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.543085 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543032    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 15 10:45:34.543151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543085    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 15 10:45:34.543151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543103    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:45:34.543151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543135    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.543838 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543825    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.543914 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543855    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.543914 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.543869    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.544108 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.544095    2565 policy_none.go:49] "None policy: Start"
Jul 15 10:45:34.544180 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.544112    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 15 10:45:34.544180 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.544123    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 15 10:45:34.545150 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.545108    2565 policy_none.go:47] "Start"
Jul 15 10:45:34.586474 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.586455    2565 manager.go:337] "Starting Device Plugin manager"
Jul 15 10:45:34.586561 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.586531    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 15 10:45:34.586561 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.586546    2565 server.go:87] "Starting device plugin registration server"
Jul 15 10:45:34.586833 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.586821    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 15 10:45:34.586870 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.586836    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 15 10:45:34.586946 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.586928    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 15 10:45:34.587092 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.587005    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 15 10:45:34.587092 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.587018    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 15 10:45:34.587586 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.587564    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 15 10:45:34.587666 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.587623    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:34.667400 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.667364    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 15 10:45:34.668427 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.667408    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 15 10:45:34.668427 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.667429    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 15 10:45:34.668427 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.667460    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 15 10:45:34.671479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.671450    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:34.686975 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.686955    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.687950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.687894    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.687950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.687930    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.687950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.687946    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.688124 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.687978    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.702817 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.702793    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.702915 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.702823    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-92.ec2.internal\": node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:34.755082 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.755032    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:34.767714 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.767688    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"]
Jul 15 10:45:34.767823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.767790    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.769354 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.769337    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.769425 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.769368    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.769425 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.769378    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.770609 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.770596    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.770747 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.770733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.770808 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.770767    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.771333 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771317    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.771419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771342    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.771419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771323    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.771419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771357    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.771419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771374    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.771419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.771386    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.772782 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.772768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.772824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.772794    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:34.773447 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.773426    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:34.773498 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.773463    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:34.773498 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.773479    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:34.793639 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.793618    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-92.ec2.internal\" not found" node="ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.798150 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.798136    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-92.ec2.internal\" not found" node="ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.826606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.826578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3e59195d56b4d838109b2802c151a5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"7f3e59195d56b4d838109b2802c151a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.826682 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.826619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.826682 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.826637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.855650 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.855628    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:34.927029 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.927132 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.927132 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3e59195d56b4d838109b2802c151a5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"7f3e59195d56b4d838109b2802c151a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.927132 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.927132 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4bd711272722a87a5d65b252b28f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"ce4bd711272722a87a5d65b252b28f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.927263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:34.927150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3e59195d56b4d838109b2802c151a5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"7f3e59195d56b4d838109b2802c151a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:34.956335 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:34.956265    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.056813 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.056762    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.095966 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.095940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:35.101697 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.101675    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:35.157965 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.157934    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.258981 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.258913    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.359280 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.359239    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.459974 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.459932    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.459974 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.459955    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 15 10:45:35.460639 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.460160    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 15 10:45:35.537969 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.537836    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 10:40:34 +0000 UTC" deadline="2028-02-21 12:27:48.490832102 +0000 UTC"
Jul 15 10:45:35.537969 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.537904    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14065h42m12.952933307s"
Jul 15 10:45:35.561104 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.561065    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.569673 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.569641    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:35.661201 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.661161    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.762137 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:35.762077    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 15 10:45:35.823677 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.823611    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:35.824602 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.824581    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:35.841316 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.841286    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:45:35.842350 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.842334    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 15 10:45:35.853659 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.853642    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:45:35.893570 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:35.893534    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3e59195d56b4d838109b2802c151a5.slice/crio-fb71366b287b64d6e9c51ec65ff256bf80ee9321f97cdb7fc17b8899767b7de1 WatchSource:0}: Error finding container fb71366b287b64d6e9c51ec65ff256bf80ee9321f97cdb7fc17b8899767b7de1: Status 404 returned error can't find the container with id fb71366b287b64d6e9c51ec65ff256bf80ee9321f97cdb7fc17b8899767b7de1
Jul 15 10:45:35.894259 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:35.894241    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4bd711272722a87a5d65b252b28f5a.slice/crio-9cf5de520882b7a94a1ed56c9f32447289087adabd37255ed9350f759b0b9324 WatchSource:0}: Error finding container 9cf5de520882b7a94a1ed56c9f32447289087adabd37255ed9350f759b0b9324: Status 404 returned error can't find the container with id 9cf5de520882b7a94a1ed56c9f32447289087adabd37255ed9350f759b0b9324
Jul 15 10:45:35.902275 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.902257    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:45:35.955431 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:35.955397    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:36.347063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.347017    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:36.504577 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.504544    2565 apiserver.go:52] "Watching apiserver"
Jul 15 10:45:36.516820 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.516789    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 15 10:45:36.518018 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.517964    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-q4lmv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq","openshift-image-registry/node-ca-2h2kf","openshift-multus/multus-additional-cni-plugins-q95q2","openshift-multus/multus-czmgj","openshift-multus/network-metrics-daemon-kkfx8","openshift-network-operator/iptables-alerter-vnggl","kube-system/global-pull-secret-syncer-lctcq","kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal","openshift-cluster-node-tuning-operator/tuned-clvsk","openshift-dns/node-resolver-ckfng","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal","openshift-network-diagnostics/network-check-target-mp8nx","openshift-ovn-kubernetes/ovnkube-node-bg9qp"]
Jul 15 10:45:36.519651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.519629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.523527 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.523505    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:36.523629 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.523577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:36.524240 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.524221    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 15 10:45:36.524453 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.524438    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.524676 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.524656    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.524792 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.524777    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 15 10:45:36.526262 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.526046    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dph4m\""
Jul 15 10:45:36.527284 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.527270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.528854 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.528829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.530289 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.530262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mr2gx\""
Jul 15 10:45:36.531237 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.530645    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 15 10:45:36.531237 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.530849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.531237 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.531063    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.532147 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.531626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.533832 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.533009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.533832 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.533091    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.534747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7mrhv\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.534840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.534754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535018    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535214    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535320    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535215    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gf6sq\""
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.535628    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4266aab-a9e6-4c6c-955a-b143ada82bf9-iptables-alerter-script\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4266aab-a9e6-4c6c-955a-b143ada82bf9-host-slash\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a5645dd-b4fd-4aff-a198-f743157e501a-serviceca\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhdh\" (UniqueName: \"kubernetes.io/projected/f4266aab-a9e6-4c6c-955a-b143ada82bf9-kube-api-access-8jhdh\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a5645dd-b4fd-4aff-a198-f743157e501a-host\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.536738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdsh\" (UniqueName: \"kubernetes.io/projected/5a5645dd-b4fd-4aff-a198-f743157e501a-kube-api-access-nfdsh\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.537694 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.536860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.538366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.539310    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.539707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.540441    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.540684    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.541017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.541476    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.541689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dgdqb\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.541907    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.542310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dkkn2\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.542546    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.542871    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.542954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.543077    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:36.543156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.543134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 15 10:45:36.543960 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.543489    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.543960 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.543728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.548633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.544996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qlf7m\""
Jul 15 10:45:36.548633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.545248    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.548633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.545343    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.548633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.545586    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 15 10:45:36.548633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.547247    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tq4mh\""
Jul 15 10:45:36.552561 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.552540    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njdkb\""
Jul 15 10:45:36.553462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.553441    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 15 10:45:36.553684 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.553650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 15 10:45:36.558084 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.558042    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:45:36.603629 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.603605    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfvbk"
Jul 15 10:45:36.620932 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.620904    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfvbk"
Jul 15 10:45:36.625672 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.625649    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 15 10:45:36.637364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdsh\" (UniqueName: \"kubernetes.io/projected/5a5645dd-b4fd-4aff-a198-f743157e501a-kube-api-access-nfdsh\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.637479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-netns\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.637479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-bin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.637479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-hostroot\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.637479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-daemon-config\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-systemd\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-kubelet\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a5645dd-b4fd-4aff-a198-f743157e501a-serviceca\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wn8l\" (UniqueName: \"kubernetes.io/projected/810a3b89-0a7c-48b6-a769-d93e1d61fe48-kube-api-access-6wn8l\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-etc-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-ovn\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-node-log\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-log-socket\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xvt\" (UniqueName: \"kubernetes.io/projected/22f1ccaa-f805-44bd-a129-148ad7244245-kube-api-access-m7xvt\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhdh\" (UniqueName: \"kubernetes.io/projected/f4266aab-a9e6-4c6c-955a-b143ada82bf9-kube-api-access-8jhdh\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a5645dd-b4fd-4aff-a198-f743157e501a-host\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.637981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.637974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-etc-kubernetes\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysconfig\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-kubernetes\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-netns\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-systemd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4266aab-a9e6-4c6c-955a-b143ada82bf9-iptables-alerter-script\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a5645dd-b4fd-4aff-a198-f743157e501a-serviceca\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-binary-copy\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.638348 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-registration-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.638750 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a5645dd-b4fd-4aff-a198-f743157e501a-host\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.638750 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-sys-fs\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.638750 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.638750 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-var-lib-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.638750 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-script-lib\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/732a12cd-410c-4422-9936-8fe6958a978b-agent-certs\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-netd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-var-lib-kubelet\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4266aab-a9e6-4c6c-955a-b143ada82bf9-iptables-alerter-script\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-tmp\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.638967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-systemd-units\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.639019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-slash\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-os-release\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-multus\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-tuned\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4266aab-a9e6-4c6c-955a-b143ada82bf9-host-slash\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-dbus\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4266aab-a9e6-4c6c-955a-b143ada82bf9-host-slash\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-kubelet\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprrb\" (UniqueName: \"kubernetes.io/projected/ab689418-71cd-41c8-9b1e-e991dbe485ae-kube-api-access-sprrb\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxsn\" (UniqueName: \"kubernetes.io/projected/9de6dbd2-8336-4020-8b28-0d4e344e9681-kube-api-access-fsxsn\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-conf-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.639506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-conf\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-system-cni-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-cnibin\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwrl\" (UniqueName: \"kubernetes.io/projected/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kube-api-access-nnwrl\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-sys\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/732a12cd-410c-4422-9936-8fe6958a978b-konnectivity-ca\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-multus-certs\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-system-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cnibin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09c549f7-a074-4a6a-a103-6517d81ce013-tmp-dir\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdwx\" (UniqueName: \"kubernetes.io/projected/09c549f7-a074-4a6a-a103-6517d81ce013-kube-api-access-wkdwx\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cni-binary-copy\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-modprobe-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-lib-modules\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09c549f7-a074-4a6a-a103-6517d81ce013-hosts-file\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.639977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-kubelet-config\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-os-release\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-run\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-host\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-bin\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-env-overrides\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-socket-dir-parent\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28n99\" (UniqueName: \"kubernetes.io/projected/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-kube-api-access-28n99\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-k8s-cni-cncf-io\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-socket-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-device-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.640924 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.641738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f1ccaa-f805-44bd-a129-148ad7244245-ovn-node-metrics-cert\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.641738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.641738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.640458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-config\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.646915 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.646889    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 15 10:45:36.652272 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.652253    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:36.652272 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.652274    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:36.652415 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.652285    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:36.652415 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.652366    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:37.152325586 +0000 UTC m=+2.987070021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:36.654888 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.654861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdsh\" (UniqueName: \"kubernetes.io/projected/5a5645dd-b4fd-4aff-a198-f743157e501a-kube-api-access-nfdsh\") pod \"node-ca-2h2kf\" (UID: \"5a5645dd-b4fd-4aff-a198-f743157e501a\") " pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.664885 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.664853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhdh\" (UniqueName: \"kubernetes.io/projected/f4266aab-a9e6-4c6c-955a-b143ada82bf9-kube-api-access-8jhdh\") pod \"iptables-alerter-vnggl\" (UID: \"f4266aab-a9e6-4c6c-955a-b143ada82bf9\") " pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.672975 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.672926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" event={"ID":"7f3e59195d56b4d838109b2802c151a5","Type":"ContainerStarted","Data":"fb71366b287b64d6e9c51ec65ff256bf80ee9321f97cdb7fc17b8899767b7de1"}
Jul 15 10:45:36.674507 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.674478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"ce4bd711272722a87a5d65b252b28f5a","Type":"ContainerStarted","Data":"9cf5de520882b7a94a1ed56c9f32447289087adabd37255ed9350f759b0b9324"}
Jul 15 10:45:36.741239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-host\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.741239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-bin\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-env-overrides\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-host\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.741239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-socket-dir-parent\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-bin\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28n99\" (UniqueName: \"kubernetes.io/projected/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-kube-api-access-28n99\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-socket-dir-parent\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-k8s-cni-cncf-io\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-socket-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-device-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-k8s-cni-cncf-io\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f1ccaa-f805-44bd-a129-148ad7244245-ovn-node-metrics-cert\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-device-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.741644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-config\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-env-overrides\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-socket-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-netns\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-bin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-netns\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-hostroot\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-bin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-daemon-config\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-systemd\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-kubelet\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wn8l\" (UniqueName: \"kubernetes.io/projected/810a3b89-0a7c-48b6-a769-d93e1d61fe48-kube-api-access-6wn8l\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-etc-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.741990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-ovn\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-node-log\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-log-socket\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xvt\" (UniqueName: \"kubernetes.io/projected/22f1ccaa-f805-44bd-a129-148ad7244245-kube-api-access-m7xvt\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.742568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-etc-kubernetes\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysconfig\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-kubernetes\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-config\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-netns\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-systemd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-run-netns\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-binary-copy\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-systemd\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-daemon-config\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-hostroot\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-ovn\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-node-log\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-log-socket\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-etc-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-kubernetes\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.743489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-binary-copy\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysconfig\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-etc-kubernetes\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-registration-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-systemd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-sys-fs\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-kubelet\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-sys-fs\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-registration-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-var-lib-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-var-lib-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.742986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-script-lib\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.742985    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/732a12cd-410c-4422-9936-8fe6958a978b-agent-certs\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.743057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:37.243034743 +0000 UTC m=+3.077779195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.744424 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-netd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-var-lib-kubelet\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-tmp\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-systemd-units\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-slash\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-cni-netd\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-os-release\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.743343    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-multus\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-tuned\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:36.743428    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:45:37.24338342 +0000 UTC m=+3.078127877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-dbus\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-kubelet\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprrb\" (UniqueName: \"kubernetes.io/projected/ab689418-71cd-41c8-9b1e-e991dbe485ae-kube-api-access-sprrb\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxsn\" (UniqueName: \"kubernetes.io/projected/9de6dbd2-8336-4020-8b28-0d4e344e9681-kube-api-access-fsxsn\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.744957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-conf-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-conf\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22f1ccaa-f805-44bd-a129-148ad7244245-ovnkube-script-lib\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-var-lib-kubelet\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-system-cni-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-run-openvswitch\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-cnibin\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-systemd-units\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22f1ccaa-f805-44bd-a129-148ad7244245-host-slash\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-dbus\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-kubelet\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-system-cni-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-cnibin\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-var-lib-cni-multus\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-os-release\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.745516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-multus-conf-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.743938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-sysctl-conf\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwrl\" (UniqueName: \"kubernetes.io/projected/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kube-api-access-nnwrl\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-sys\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/732a12cd-410c-4422-9936-8fe6958a978b-konnectivity-ca\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-multus-certs\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-sys\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9de6dbd2-8336-4020-8b28-0d4e344e9681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-system-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-system-cni-dir\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cnibin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09c549f7-a074-4a6a-a103-6517d81ce013-tmp-dir\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cnibin\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdwx\" (UniqueName: \"kubernetes.io/projected/09c549f7-a074-4a6a-a103-6517d81ce013-kube-api-access-wkdwx\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-host-run-multus-certs\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cni-binary-copy\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-modprobe-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-lib-modules\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09c549f7-a074-4a6a-a103-6517d81ce013-hosts-file\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-kubelet-config\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-os-release\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-run\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09c549f7-a074-4a6a-a103-6517d81ce013-tmp-dir\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-run\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09c549f7-a074-4a6a-a103-6517d81ce013-hosts-file\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-lib-modules\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/732a12cd-410c-4422-9936-8fe6958a978b-konnectivity-ca\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24527bdc-899a-498d-ae7c-8c09a08e921d-kubelet-config\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-os-release\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f1ccaa-f805-44bd-a129-148ad7244245-ovn-node-metrics-cert\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.744971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-modprobe-d\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.746787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.745040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9de6dbd2-8336-4020-8b28-0d4e344e9681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.747558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.745243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-cni-binary-copy\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.747558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.746093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/732a12cd-410c-4422-9936-8fe6958a978b-agent-certs\") pod \"konnectivity-agent-q4lmv\" (UID: \"732a12cd-410c-4422-9936-8fe6958a978b\") " pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.747558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.746881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-etc-tuned\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.749757 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.749728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab689418-71cd-41c8-9b1e-e991dbe485ae-tmp\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.758119 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.758092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xvt\" (UniqueName: \"kubernetes.io/projected/22f1ccaa-f805-44bd-a129-148ad7244245-kube-api-access-m7xvt\") pod \"ovnkube-node-bg9qp\" (UID: \"22f1ccaa-f805-44bd-a129-148ad7244245\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.762715 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.762688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxsn\" (UniqueName: \"kubernetes.io/projected/9de6dbd2-8336-4020-8b28-0d4e344e9681-kube-api-access-fsxsn\") pod \"multus-additional-cni-plugins-q95q2\" (UID: \"9de6dbd2-8336-4020-8b28-0d4e344e9681\") " pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.765445 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.765418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwrl\" (UniqueName: \"kubernetes.io/projected/1e5fd44e-5d83-4c58-bd16-26f63fb032b8-kube-api-access-nnwrl\") pod \"aws-ebs-csi-driver-node-8c6hq\" (UID: \"1e5fd44e-5d83-4c58-bd16-26f63fb032b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.765564 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.765541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28n99\" (UniqueName: \"kubernetes.io/projected/9cdb8adc-6c5a-487d-ae85-49f1fabbaa17-kube-api-access-28n99\") pod \"multus-czmgj\" (UID: \"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17\") " pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.766349 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.766322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprrb\" (UniqueName: \"kubernetes.io/projected/ab689418-71cd-41c8-9b1e-e991dbe485ae-kube-api-access-sprrb\") pod \"tuned-clvsk\" (UID: \"ab689418-71cd-41c8-9b1e-e991dbe485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.766621 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.766586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wn8l\" (UniqueName: \"kubernetes.io/projected/810a3b89-0a7c-48b6-a769-d93e1d61fe48-kube-api-access-6wn8l\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:36.766840 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.766814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdwx\" (UniqueName: \"kubernetes.io/projected/09c549f7-a074-4a6a-a103-6517d81ce013-kube-api-access-wkdwx\") pod \"node-resolver-ckfng\" (UID: \"09c549f7-a074-4a6a-a103-6517d81ce013\") " pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:36.844828 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.844788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnggl"
Jul 15 10:45:36.853196 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.853167    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2h2kf"
Jul 15 10:45:36.855287 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:36.855211    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4266aab_a9e6_4c6c_955a_b143ada82bf9.slice/crio-69c938cbb28c814ef00780f54b3b754f968f9b40fa8e110313bc1e16aa2321f0 WatchSource:0}: Error finding container 69c938cbb28c814ef00780f54b3b754f968f9b40fa8e110313bc1e16aa2321f0: Status 404 returned error can't find the container with id 69c938cbb28c814ef00780f54b3b754f968f9b40fa8e110313bc1e16aa2321f0
Jul 15 10:45:36.863427 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.863396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q95q2"
Jul 15 10:45:36.864607 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:36.864563    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5645dd_b4fd_4aff_a198_f743157e501a.slice/crio-e0dd83cda8c0293ce0a95b8635f35537b8260c0059c6935dfe5468018308e5e2 WatchSource:0}: Error finding container e0dd83cda8c0293ce0a95b8635f35537b8260c0059c6935dfe5468018308e5e2: Status 404 returned error can't find the container with id e0dd83cda8c0293ce0a95b8635f35537b8260c0059c6935dfe5468018308e5e2
Jul 15 10:45:36.870808 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.870783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czmgj"
Jul 15 10:45:36.873176 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:36.873029    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de6dbd2_8336_4020_8b28_0d4e344e9681.slice/crio-6ba1c037a75c15ebbc5315e374c629e2117bfd3d206f367789289990ba078ed2 WatchSource:0}: Error finding container 6ba1c037a75c15ebbc5315e374c629e2117bfd3d206f367789289990ba078ed2: Status 404 returned error can't find the container with id 6ba1c037a75c15ebbc5315e374c629e2117bfd3d206f367789289990ba078ed2
Jul 15 10:45:36.880819 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:36.880789    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdb8adc_6c5a_487d_ae85_49f1fabbaa17.slice/crio-15ffa2ad54c084777f2de21ec20c7cdbb0652ea14d407c54a1450159cbaf25ae WatchSource:0}: Error finding container 15ffa2ad54c084777f2de21ec20c7cdbb0652ea14d407c54a1450159cbaf25ae: Status 404 returned error can't find the container with id 15ffa2ad54c084777f2de21ec20c7cdbb0652ea14d407c54a1450159cbaf25ae
Jul 15 10:45:36.880929 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.880834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:45:36.889802 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.889778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:36.897634 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.897609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq"
Jul 15 10:45:36.907432 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.907409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clvsk"
Jul 15 10:45:36.913265 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:36.913239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ckfng"
Jul 15 10:45:37.248515 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.248422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:37.248515 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.248482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:37.248515 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.248503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248594    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248608    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248649    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:45:38.248635979 +0000 UTC m=+4.083380409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248677    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:38.24865658 +0000 UTC m=+4.083401024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248615    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248703    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248717    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:37.248801 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.248764    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:38.248752882 +0000 UTC m=+4.083497313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:37.623705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.623616    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:40:36 +0000 UTC" deadline="2028-03-30 13:34:15.174320488 +0000 UTC"
Jul 15 10:45:37.623705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.623653    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14978h48m37.550673065s"
Jul 15 10:45:37.668460 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.668429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:37.668653 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:37.668555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:37.681117 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.681085    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnggl" event={"ID":"f4266aab-a9e6-4c6c-955a-b143ada82bf9","Type":"ContainerStarted","Data":"69c938cbb28c814ef00780f54b3b754f968f9b40fa8e110313bc1e16aa2321f0"}
Jul 15 10:45:37.682284 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.682261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czmgj" event={"ID":"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17","Type":"ContainerStarted","Data":"15ffa2ad54c084777f2de21ec20c7cdbb0652ea14d407c54a1450159cbaf25ae"}
Jul 15 10:45:37.683291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.683256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerStarted","Data":"6ba1c037a75c15ebbc5315e374c629e2117bfd3d206f367789289990ba078ed2"}
Jul 15 10:45:37.685802 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:37.685776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2h2kf" event={"ID":"5a5645dd-b4fd-4aff-a198-f743157e501a","Type":"ContainerStarted","Data":"e0dd83cda8c0293ce0a95b8635f35537b8260c0059c6935dfe5468018308e5e2"}
Jul 15 10:45:38.001322 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:38.001169    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1ccaa_f805_44bd_a129_148ad7244245.slice/crio-c148e349932212e3f8abc07642f61db68e61e887e95496a5858d8ab4f537687d WatchSource:0}: Error finding container c148e349932212e3f8abc07642f61db68e61e887e95496a5858d8ab4f537687d: Status 404 returned error can't find the container with id c148e349932212e3f8abc07642f61db68e61e887e95496a5858d8ab4f537687d
Jul 15 10:45:38.006195 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:38.006107    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c549f7_a074_4a6a_a103_6517d81ce013.slice/crio-63ebace4f33bac35f672fb7fb9ba2919fd2f799ad3b1182c64f5f422622cdab0 WatchSource:0}: Error finding container 63ebace4f33bac35f672fb7fb9ba2919fd2f799ad3b1182c64f5f422622cdab0: Status 404 returned error can't find the container with id 63ebace4f33bac35f672fb7fb9ba2919fd2f799ad3b1182c64f5f422622cdab0
Jul 15 10:45:38.009315 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:38.009232    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732a12cd_410c_4422_9936_8fe6958a978b.slice/crio-c64ca5122ee2440d49a0fd1d1598e481d4d910c5e6a3548b27713f816a3b5a0f WatchSource:0}: Error finding container c64ca5122ee2440d49a0fd1d1598e481d4d910c5e6a3548b27713f816a3b5a0f: Status 404 returned error can't find the container with id c64ca5122ee2440d49a0fd1d1598e481d4d910c5e6a3548b27713f816a3b5a0f
Jul 15 10:45:38.012153 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:38.010235    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab689418_71cd_41c8_9b1e_e991dbe485ae.slice/crio-02964c966c6d7af8a05a39dd6289b542693da46d3abed5bd87e12dcb7605248a WatchSource:0}: Error finding container 02964c966c6d7af8a05a39dd6289b542693da46d3abed5bd87e12dcb7605248a: Status 404 returned error can't find the container with id 02964c966c6d7af8a05a39dd6289b542693da46d3abed5bd87e12dcb7605248a
Jul 15 10:45:38.017192 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:45:38.017148    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5fd44e_5d83_4c58_bd16_26f63fb032b8.slice/crio-e93f63036535c806b1b11c3aa5b6545bd6a33bb5350f09e54b1bd0041c0c2d10 WatchSource:0}: Error finding container e93f63036535c806b1b11c3aa5b6545bd6a33bb5350f09e54b1bd0041c0c2d10: Status 404 returned error can't find the container with id e93f63036535c806b1b11c3aa5b6545bd6a33bb5350f09e54b1bd0041c0c2d10
Jul 15 10:45:38.258789 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.258445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:38.258789 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.258759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:38.258789 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258670    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258822    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258839    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258878    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258907    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.258886295 +0000 UTC m=+6.093630731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258932    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.258915319 +0000 UTC m=+6.093659755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.258987    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.259021    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.259008831 +0000 UTC m=+6.093753267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:38.259095 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.258790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:38.624150 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.623982    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:40:36 +0000 UTC" deadline="2028-01-02 22:03:29.753222786 +0000 UTC"
Jul 15 10:45:38.624150 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.624047    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12875h17m51.12917979s"
Jul 15 10:45:38.671570 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.670748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:38.671570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.670885    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:38.671570 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.671361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:38.671570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:38.671473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:38.697178 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.697134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q4lmv" event={"ID":"732a12cd-410c-4422-9936-8fe6958a978b","Type":"ContainerStarted","Data":"c64ca5122ee2440d49a0fd1d1598e481d4d910c5e6a3548b27713f816a3b5a0f"}
Jul 15 10:45:38.702075 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.702015    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce4bd711272722a87a5d65b252b28f5a" containerID="233c57bd6d9cf5a8aa73282858cff01b830d995ef2b266c6254d420411af52ac" exitCode=0
Jul 15 10:45:38.702248 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.702103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"ce4bd711272722a87a5d65b252b28f5a","Type":"ContainerDied","Data":"233c57bd6d9cf5a8aa73282858cff01b830d995ef2b266c6254d420411af52ac"}
Jul 15 10:45:38.708858 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.708830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq" event={"ID":"1e5fd44e-5d83-4c58-bd16-26f63fb032b8","Type":"ContainerStarted","Data":"e93f63036535c806b1b11c3aa5b6545bd6a33bb5350f09e54b1bd0041c0c2d10"}
Jul 15 10:45:38.714973 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.714905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ckfng" event={"ID":"09c549f7-a074-4a6a-a103-6517d81ce013","Type":"ContainerStarted","Data":"63ebace4f33bac35f672fb7fb9ba2919fd2f799ad3b1182c64f5f422622cdab0"}
Jul 15 10:45:38.718792 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.718759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"c148e349932212e3f8abc07642f61db68e61e887e95496a5858d8ab4f537687d"}
Jul 15 10:45:38.728234 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.728203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" event={"ID":"7f3e59195d56b4d838109b2802c151a5","Type":"ContainerStarted","Data":"58a3140542e31e057a70a61aefd707921acc23ccf1512d53ea69fc899e73a83b"}
Jul 15 10:45:38.742264 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:38.742206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clvsk" event={"ID":"ab689418-71cd-41c8-9b1e-e991dbe485ae","Type":"ContainerStarted","Data":"02964c966c6d7af8a05a39dd6289b542693da46d3abed5bd87e12dcb7605248a"}
Jul 15 10:45:39.669450 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:39.668904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:39.669450 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:39.669047    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:39.756806 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:39.756766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"ce4bd711272722a87a5d65b252b28f5a","Type":"ContainerStarted","Data":"c580a8938d8e7cc17e35577a464a645dd6a96a4cd3835486f18e86254b29e1f8"}
Jul 15 10:45:39.776951 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:39.776875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" podStartSLOduration=3.776853664 podStartE2EDuration="3.776853664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:38.762396627 +0000 UTC m=+4.597141135" watchObservedRunningTime="2026-07-15 10:45:39.776853664 +0000 UTC m=+5.611598121"
Jul 15 10:45:40.274456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:40.274412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:40.274675 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:40.274498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:40.274675 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:40.274528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:40.274675 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.274660    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:40.274847 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.274727    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:45:44.274707573 +0000 UTC m=+10.109452006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:40.275204 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275181    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:40.275301 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275210    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:40.275301 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275225    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:40.275301 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275277    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:44.275260484 +0000 UTC m=+10.110004922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:40.275476 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275346    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:40.275476 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.275378    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:44.275367723 +0000 UTC m=+10.110112159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:40.668329 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:40.667626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:40.668329 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.667768    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:40.668329 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:40.668149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:40.668329 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:40.668245    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:41.669100 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:41.668601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:41.669100 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:41.668749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:42.670145 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:42.669363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:42.670145 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:42.669449    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:42.670783 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:42.670629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:42.670783 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:42.670736    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:43.667971 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:43.667849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:43.667971 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:43.667953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:44.307434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:44.307398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:44.307453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:44.307521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307562    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307635    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:52.307613322 +0000 UTC m=+18.142357756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307662    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307680    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307692    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307731    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:52.307720499 +0000 UTC m=+18.142464926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307783    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:44.307889 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.307806    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:45:52.307798789 +0000 UTC m=+18.142543217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:44.668731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:44.668651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:44.668731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:44.668693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:44.668939 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.668785    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:44.668993 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:44.668934    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:45.668456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:45.668422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:45.668870 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:45.668566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:46.667726 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:46.667700    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:46.667917 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:46.667814    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:46.667917 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:46.667859    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:46.668032 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:46.667937    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:47.668076 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:47.668026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:47.668501 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:47.668174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:48.668019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:48.667979    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:48.668217 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:48.668128    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:48.668660 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:48.668627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:48.668781 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:48.668759    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:49.668228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:49.668195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:49.668661 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:49.668331    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:50.668027 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:50.667990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:50.668204 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:50.667990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:50.668204 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:50.668166    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:50.668204 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:50.668189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:51.668775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:51.668215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:51.668775 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:51.668345    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:52.368322 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:52.368282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:52.368322 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:52.368316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:52.368351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368458    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368472    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368486    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368495    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368463    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368538    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:08.368517249 +0000 UTC m=+34.203261691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:52.368570 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368563    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:46:08.368547335 +0000 UTC m=+34.203291764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:52.368979 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.368580    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:08.368571157 +0000 UTC m=+34.203315588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:52.668007 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:52.667924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:52.668176 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.668066    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:52.668176 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:52.668107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:52.668317 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:52.668230    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:53.667849 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:53.667802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:53.668314 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:53.667927    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:54.670022 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:54.669535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:54.670022 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:54.669655    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:54.670022 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:54.669663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:54.670022 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:54.669776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:55.667594 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:55.667559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:55.667760 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:55.667690    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:56.668470 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.668438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:56.668814 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.668469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:56.668814 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:56.668589    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:56.669164 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:56.669029    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:56.797329 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.796985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2h2kf" event={"ID":"5a5645dd-b4fd-4aff-a198-f743157e501a","Type":"ContainerStarted","Data":"5a88c4ac8fc1588118c11161579b6e0a81468df43a3ce3b7353008ae5cdb63b4"}
Jul 15 10:45:56.800531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.799032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clvsk" event={"ID":"ab689418-71cd-41c8-9b1e-e991dbe485ae","Type":"ContainerStarted","Data":"62b58e32139094cb713cf65d59f0654fb75e23a78e134ef32067300378da892f"}
Jul 15 10:45:56.801585 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.801311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q4lmv" event={"ID":"732a12cd-410c-4422-9936-8fe6958a978b","Type":"ContainerStarted","Data":"d3423204477b3090e46c8d5c16fd67c44c907c339d0756450bbd5a6379b15eb7"}
Jul 15 10:45:56.803075 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.802896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czmgj" event={"ID":"9cdb8adc-6c5a-487d-ae85-49f1fabbaa17","Type":"ContainerStarted","Data":"e7f6876dd2e05a233ab0ee46cadd6b8ada4608c3581baf386c3d4531abea66a1"}
Jul 15 10:45:56.804184 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.804161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq" event={"ID":"1e5fd44e-5d83-4c58-bd16-26f63fb032b8","Type":"ContainerStarted","Data":"f5eb584983930a902d3f7362ade468e167cbd8e0b16123435977140b8d1ddf75"}
Jul 15 10:45:56.805331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.805310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ckfng" event={"ID":"09c549f7-a074-4a6a-a103-6517d81ce013","Type":"ContainerStarted","Data":"739338b008fa8bbffb27bd41d6679250c5376647408923f7f98695c2d978257d"}
Jul 15 10:45:56.806299 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.806280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerStarted","Data":"64dcdb82cff7b8a8b5c3308de7ce16b48d7523fc3bc029a800aedc5b640dc77a"}
Jul 15 10:45:56.813076 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.813011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" podStartSLOduration=20.813000322 podStartE2EDuration="20.813000322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:39.77807432 +0000 UTC m=+5.612818776" watchObservedRunningTime="2026-07-15 10:45:56.813000322 +0000 UTC m=+22.647744771"
Jul 15 10:45:56.813487 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.813455    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2h2kf" podStartSLOduration=6.884569283 podStartE2EDuration="21.813447752s" totalImagesPullingTime="14.928878469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:56.812956587 +0000 UTC m=+22.647701037" watchObservedRunningTime="2026-07-15 10:45:56.813447752 +0000 UTC m=+22.648192202"
Jul 15 10:45:56.858896 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.858836    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-czmgj" podStartSLOduration=2.17792864 podStartE2EDuration="21.858815889s" totalImagesPullingTime="19.680887249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:56.857969821 +0000 UTC m=+22.692714271" watchObservedRunningTime="2026-07-15 10:45:56.858815889 +0000 UTC m=+22.693560338"
Jul 15 10:45:56.873117 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.873067    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q4lmv" podStartSLOduration=3.35959498 podStartE2EDuration="21.873027375s" totalImagesPullingTime="18.513432395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:56.872751565 +0000 UTC m=+22.707496034" watchObservedRunningTime="2026-07-15 10:45:56.873027375 +0000 UTC m=+22.707771824"
Jul 15 10:45:56.911034 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.910972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-clvsk" podStartSLOduration=3.375131841 podStartE2EDuration="21.910951031s" totalImagesPullingTime="18.53581919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:56.894017647 +0000 UTC m=+22.728762118" watchObservedRunningTime="2026-07-15 10:45:56.910951031 +0000 UTC m=+22.745695482"
Jul 15 10:45:56.911212 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:56.911179    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ckfng" podStartSLOduration=3.368488226 podStartE2EDuration="21.911165857s" totalImagesPullingTime="18.542677631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:56.910893057 +0000 UTC m=+22.745637507" watchObservedRunningTime="2026-07-15 10:45:56.911165857 +0000 UTC m=+22.745910484"
Jul 15 10:45:57.399958 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.399705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:57.400393 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.400378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:57.499744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.499714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:45:57.593131 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.593099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:45:57.668415 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.668393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:57.668542 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:57.668475    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:57.811640 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.811614    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:45:57.811950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.811928    2565 generic.go:356] "Generic (PLEG): container finished" podID="22f1ccaa-f805-44bd-a129-148ad7244245" containerID="568c09c4cf5b6a08461e12d839d3909b541aa428e1883bade0e92a392d058e8d" exitCode=1
Jul 15 10:45:57.812036 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"5079de91cf23f3ad3f0ca08d952ce9644d3c85a2379006c83ebf04c4b6aba5d5"}
Jul 15 10:45:57.812143 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"dd3b8105c01516797f86c217b2269c20ea6d3c8bbe46e26c6e334ac8d5bc1dc1"}
Jul 15 10:45:57.812193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"b4bb5eeae79cf87bef9a6ab31b5e677824007ccafcc7d950ba5e4ca69ba2311f"}
Jul 15 10:45:57.812246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"ac3e839747d598bf003ade90f7555006e3ba908b431cd885626a52073840b2da"}
Jul 15 10:45:57.812297 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerDied","Data":"568c09c4cf5b6a08461e12d839d3909b541aa428e1883bade0e92a392d058e8d"}
Jul 15 10:45:57.812297 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.812293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"9b66c0cde42978aadbfe5216e5b2827a2fbb02966f1fd91533ff5e8516c94572"}
Jul 15 10:45:57.813476 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.813453    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="64dcdb82cff7b8a8b5c3308de7ce16b48d7523fc3bc029a800aedc5b640dc77a" exitCode=0
Jul 15 10:45:57.813550 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.813519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"64dcdb82cff7b8a8b5c3308de7ce16b48d7523fc3bc029a800aedc5b640dc77a"}
Jul 15 10:45:57.814913 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.814883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnggl" event={"ID":"f4266aab-a9e6-4c6c-955a-b143ada82bf9","Type":"ContainerStarted","Data":"81aff65e9fd531fd8708909d67b0b2dfe5e33c1a003a5bd4a6f5dc4b86a9cf0a"}
Jul 15 10:45:57.815100 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.815085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:57.815557 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.815538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q4lmv"
Jul 15 10:45:57.883316 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.883226    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vnggl" podStartSLOduration=3.213604709 podStartE2EDuration="22.883207565s" totalImagesPullingTime="19.669602856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:57.882757846 +0000 UTC m=+23.717502295" watchObservedRunningTime="2026-07-15 10:45:57.883207565 +0000 UTC m=+23.717952017"
Jul 15 10:45:57.917313 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:57.917283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:45:58.070648 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.070614    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:45:58.222901 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.222764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:45:58.671423 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.671382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:45:58.671791 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:58.671487    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:45:58.671791 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.671704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:45:58.671905 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:58.671816    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:45:58.769777 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.769649    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 15 10:45:58.819469 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:58.819434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq" event={"ID":"1e5fd44e-5d83-4c58-bd16-26f63fb032b8","Type":"ContainerStarted","Data":"77a28a69165f94d6bcda2c4ecf64cdca57abc4a85418de355987bb65397c72a6"}
Jul 15 10:45:59.598922 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.598778    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T10:45:58.769667922Z","UUID":"363368db-651d-4bdd-aaa1-1599d66fe4bc","Handler":null,"Name":"","Endpoint":""}
Jul 15 10:45:59.600961 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.600941    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 15 10:45:59.601103 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.600986    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 15 10:45:59.667666 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.667630    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:45:59.667816 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:45:59.667757    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:45:59.825266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.825242    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:45:59.825745 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:45:59.825674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"e470a71cdd04220ec3911083567093ea9e218b1e1991d1d957f59c6351daed22"}
Jul 15 10:46:00.667919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:00.667690    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:00.668137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:00.667777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:00.668137 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:00.668028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:46:00.668137 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:00.668111    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:46:00.830894 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:00.830859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq" event={"ID":"1e5fd44e-5d83-4c58-bd16-26f63fb032b8","Type":"ContainerStarted","Data":"a3b64cb94249e39ab112b44d5ce2b0ef54fe53430cd224eef21677a56328cf92"}
Jul 15 10:46:00.853086 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:00.853010    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8c6hq" podStartSLOduration=4.2995775179999995 podStartE2EDuration="25.852989111s" totalImagesPullingTime="21.553411593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:00.852341738 +0000 UTC m=+26.687086189" watchObservedRunningTime="2026-07-15 10:46:00.852989111 +0000 UTC m=+26.687733562"
Jul 15 10:46:00.974321 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:00.974236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:01.077489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:01.077442    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:01.668730 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:01.668684    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:01.668934 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:01.668821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:46:01.918846 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:01.918819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:02.019869 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.019838    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:02.542353 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.542122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:02.635992 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.635952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:02.668314 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.668285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:02.668423 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.668287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:02.668423 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:02.668408    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:46:02.668512 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:02.668489    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:46:02.837593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.837570    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:46:02.837950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.837925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"db4e99a731dca38d2ad7205794b1d82407fc8900ef537012379b8a12efc55b64"}
Jul 15 10:46:02.838199 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.838176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:02.838339 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.838248    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:02.838440 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.838418    2565 scope.go:117] "RemoveContainer" containerID="568c09c4cf5b6a08461e12d839d3909b541aa428e1883bade0e92a392d058e8d"
Jul 15 10:46:02.839952 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.839927    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="daebe6dfb7d32034098dac7be29eba45a31f50d84c4a244ec544ab7847ce4964" exitCode=0
Jul 15 10:46:02.840064 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.839986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"daebe6dfb7d32034098dac7be29eba45a31f50d84c4a244ec544ab7847ce4964"}
Jul 15 10:46:02.855644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.855619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:02.861492 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:02.861471    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:03.668228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:03.668197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:03.668736 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:03.668303    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:46:03.845131 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:03.845105    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:46:03.845470 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:03.845443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" event={"ID":"22f1ccaa-f805-44bd-a129-148ad7244245","Type":"ContainerStarted","Data":"5a40aa0d2cc8db6412623b56dcfa01c16576fce29239dcb382955364f6e5f08c"}
Jul 15 10:46:03.845744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:03.845724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:03.881231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:03.881179    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp" podStartSLOduration=10.100686808 podStartE2EDuration="28.881164118s" totalImagesPullingTime="18.78047731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:03.879425503 +0000 UTC m=+29.714169953" watchObservedRunningTime="2026-07-15 10:46:03.881164118 +0000 UTC m=+29.715908568"
Jul 15 10:46:04.309084 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.308981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lctcq"]
Jul 15 10:46:04.309248 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.309107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:04.309248 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:04.309211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:46:04.312398 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.312364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp8nx"]
Jul 15 10:46:04.312542 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.312476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:04.312607 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:04.312554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:46:04.312904 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.312880    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkfx8"]
Jul 15 10:46:04.313020 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.312978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:04.313108 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:04.313078    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:46:04.850285 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.850247    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="e4959af6f89982ee517bbb2b67eb28da8338102460d749f859b39ace5d00c2ec" exitCode=0
Jul 15 10:46:04.850790 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:04.850328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"e4959af6f89982ee517bbb2b67eb28da8338102460d749f859b39ace5d00c2ec"}
Jul 15 10:46:06.668232 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:06.668058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:06.668702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:06.668060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:06.668702 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:06.668335    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:46:06.668702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:06.668096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:06.668702 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:06.668430    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:46:06.668702 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:06.668550    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:46:07.858502 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:07.858465    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="8da98986e0852895c187c1685519b1994f10795cf1b84cadcb6ad4ccb07b5cf2" exitCode=0
Jul 15 10:46:07.858918 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:07.858509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"8da98986e0852895c187c1685519b1994f10795cf1b84cadcb6ad4ccb07b5cf2"}
Jul 15 10:46:08.381359 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.381319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:08.381359 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.381354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.381409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381481    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381514    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381528    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381536    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4ksj for pod openshift-network-diagnostics/network-check-target-mp8nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381546    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:40.381532013 +0000 UTC m=+66.216276441 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381573    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj podName:96cf6719-f021-43f0-9eeb-d17c1f893325 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:40.381562499 +0000 UTC m=+66.216306926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4ksj" (UniqueName: "kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj") pod "network-check-target-mp8nx" (UID: "96cf6719-f021-43f0-9eeb-d17c1f893325") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:08.381581 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381481    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:08.381822 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.381622    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret podName:24527bdc-899a-498d-ae7c-8c09a08e921d nodeName:}" failed. No retries permitted until 2026-07-15 10:46:40.381611116 +0000 UTC m=+66.216355547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret") pod "global-pull-secret-syncer-lctcq" (UID: "24527bdc-899a-498d-ae7c-8c09a08e921d") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:08.605623 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.605590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:08.668532 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.668452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:08.668532 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.668471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:08.668754 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.668455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:08.668754 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.668555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lctcq" podUID="24527bdc-899a-498d-ae7c-8c09a08e921d"
Jul 15 10:46:08.668754 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.668636    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkfx8" podUID="810a3b89-0a7c-48b6-a769-d93e1d61fe48"
Jul 15 10:46:08.668754 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:08.668680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8nx" podUID="96cf6719-f021-43f0-9eeb-d17c1f893325"
Jul 15 10:46:08.844783 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:08.844753    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:09.516302 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.516270    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeReady"
Jul 15 10:46:09.516819 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.516396    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 15 10:46:09.605087 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.605034    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rztx6"]
Jul 15 10:46:09.607035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.607013    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xv7x9"]
Jul 15 10:46:09.607178 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.607155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.609073 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.609036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.609784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.609765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 15 10:46:09.610371 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.610025    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 15 10:46:09.610371 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.610288    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s2xwp\""
Jul 15 10:46:09.610371 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.610341    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 15 10:46:09.611033 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.611015    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8cmqw\""
Jul 15 10:46:09.611353 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.611333    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 15 10:46:09.611631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.611617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 15 10:46:09.638893 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.638867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv7x9"]
Jul 15 10:46:09.680149 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.680114    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rztx6"]
Jul 15 10:46:09.692371 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd6d\" (UniqueName: \"kubernetes.io/projected/c21dea71-cff4-4203-95e7-b1e5112fa9d2-kube-api-access-wkd6d\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.692543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-cert\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.692543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c21dea71-cff4-4203-95e7-b1e5112fa9d2-config-volume\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.692543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21dea71-cff4-4203-95e7-b1e5112fa9d2-tmp-dir\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.692708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.692708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.692622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lj2\" (UniqueName: \"kubernetes.io/projected/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-kube-api-access-b7lj2\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.793384 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.793384 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lj2\" (UniqueName: \"kubernetes.io/projected/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-kube-api-access-b7lj2\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.793620 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd6d\" (UniqueName: \"kubernetes.io/projected/c21dea71-cff4-4203-95e7-b1e5112fa9d2-kube-api-access-wkd6d\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.793620 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-cert\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.793620 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:09.793462    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:09.793620 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:09.793524    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:10.293505637 +0000 UTC m=+36.128250082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:09.793801 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c21dea71-cff4-4203-95e7-b1e5112fa9d2-config-volume\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.793801 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21dea71-cff4-4203-95e7-b1e5112fa9d2-tmp-dir\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.794014 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.793969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21dea71-cff4-4203-95e7-b1e5112fa9d2-tmp-dir\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.794160 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.794121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c21dea71-cff4-4203-95e7-b1e5112fa9d2-config-volume\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.798650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.798627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-cert\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.805893 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.805866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lj2\" (UniqueName: \"kubernetes.io/projected/b55bedd1-7fc7-4595-a8fa-7b5801ae43ee-kube-api-access-b7lj2\") pod \"ingress-canary-rztx6\" (UID: \"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee\") " pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:09.806110 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.806092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd6d\" (UniqueName: \"kubernetes.io/projected/c21dea71-cff4-4203-95e7-b1e5112fa9d2-kube-api-access-wkd6d\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:09.920152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:09.920118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rztx6"
Jul 15 10:46:10.074785 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.074597    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rztx6"]
Jul 15 10:46:10.079549 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:10.079515    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55bedd1_7fc7_4595_a8fa_7b5801ae43ee.slice/crio-38c51fad0ea0e3de54093c9f37190432a03d44bf03878e1e60017d4503d30e88 WatchSource:0}: Error finding container 38c51fad0ea0e3de54093c9f37190432a03d44bf03878e1e60017d4503d30e88: Status 404 returned error can't find the container with id 38c51fad0ea0e3de54093c9f37190432a03d44bf03878e1e60017d4503d30e88
Jul 15 10:46:10.297037 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.297001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:10.297240 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:10.297170    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:10.297240 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:10.297240    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:11.297221882 +0000 UTC m=+37.131966497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:10.667905 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.667870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:10.668539 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.668432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:10.668643 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.668625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:10.671664 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.671646    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfsm8\""
Jul 15 10:46:10.672143 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.672119    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:46:10.672259 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.672121    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:46:10.673478 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.673453    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhf7j\""
Jul 15 10:46:10.673576 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.673508    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:46:10.673645 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.673512    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:46:10.866063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:10.866010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rztx6" event={"ID":"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee","Type":"ContainerStarted","Data":"38c51fad0ea0e3de54093c9f37190432a03d44bf03878e1e60017d4503d30e88"}
Jul 15 10:46:11.308901 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:11.308860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:11.309097 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:11.309003    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:11.309097 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:11.309092    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:13.309072803 +0000 UTC m=+39.143817240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:11.466345 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:11.466285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:11.552546 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:11.552512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:13.325144 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:13.325109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:13.325578 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:13.325296    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:13.325578 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:13.325376    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:17.32535445 +0000 UTC m=+43.160098893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:13.920335 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:13.920284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:14.028260 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:14.028234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:14.878021 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:14.877984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rztx6" event={"ID":"b55bedd1-7fc7-4595-a8fa-7b5801ae43ee","Type":"ContainerStarted","Data":"47f004c9c74318ddb8d9076bb10d6ea198d663a69260f7b01ed2baf890092a99"}
Jul 15 10:46:14.880496 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:14.880473    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="137954775a8be67010d5ab38c484b3d8050a3c0299f1aaa4082fef70587700cb" exitCode=0
Jul 15 10:46:14.880638 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:14.880503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"137954775a8be67010d5ab38c484b3d8050a3c0299f1aaa4082fef70587700cb"}
Jul 15 10:46:14.900766 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:14.900722    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rztx6" podStartSLOduration=2.053926228 podStartE2EDuration="5.900701401s" totalImagesPullingTime="3.846775173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:14.900605557 +0000 UTC m=+40.735350022" watchObservedRunningTime="2026-07-15 10:46:14.900701401 +0000 UTC m=+40.735445851"
Jul 15 10:46:15.886481 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:15.886440    2565 generic.go:356] "Generic (PLEG): container finished" podID="9de6dbd2-8336-4020-8b28-0d4e344e9681" containerID="ad7d898e0d1c36ae78bc1352e5e00d2448140e8199b5e86c4ee8dcae38f906f4" exitCode=0
Jul 15 10:46:15.886821 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:15.886515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerDied","Data":"ad7d898e0d1c36ae78bc1352e5e00d2448140e8199b5e86c4ee8dcae38f906f4"}
Jul 15 10:46:16.892227 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:16.892193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q95q2" event={"ID":"9de6dbd2-8336-4020-8b28-0d4e344e9681","Type":"ContainerStarted","Data":"27abecdc85635565b3b5e5d19b5d720cbbb1e03d4620c56633cde93c2e95ee6c"}
Jul 15 10:46:16.925606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:16.925553    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q95q2" podStartSLOduration=8.718080396 podStartE2EDuration="41.925538976s" totalImagesPullingTime="33.20745858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:16.924138826 +0000 UTC m=+42.758883292" watchObservedRunningTime="2026-07-15 10:46:16.925538976 +0000 UTC m=+42.760283425"
Jul 15 10:46:17.230283 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:17.230198    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:17.352008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:17.351966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:17.352194 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:17.352132    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:17.352234 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:17.352198    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.352182786 +0000 UTC m=+51.186927214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:17.452628 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:17.452586    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:24.017981 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:24.017942    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:24.120864 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:24.120833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:25.407365 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:25.407320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:25.407739 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:25.407449    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:25.407739 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:25.407513    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls podName:c21dea71-cff4-4203-95e7-b1e5112fa9d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:41.407497321 +0000 UTC m=+67.242241753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls") pod "dns-default-xv7x9" (UID: "c21dea71-cff4-4203-95e7-b1e5112fa9d2") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:28.083839 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.083796    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-hc5rp"]
Jul 15 10:46:28.105719 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.105691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.108682 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.108661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 15 10:46:28.108833 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.108817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 15 10:46:28.109030 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.109014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:28.109657 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.109641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8vscr\""
Jul 15 10:46:28.109720 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.109641    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:28.113566 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.113544    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-hc5rp"]
Jul 15 10:46:28.115305 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.115268    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 15 10:46:28.175491 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.175456    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"]
Jul 15 10:46:28.195738 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.195712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b4b849546-cmq8n"]
Jul 15 10:46:28.195896 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.195841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.198171 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.198150    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 15 10:46:28.198903 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.198882    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:28.199014 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.198881    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:28.199014 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.199002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wt7gm\""
Jul 15 10:46:28.199307 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.199288    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 15 10:46:28.219641 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.219616    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"]
Jul 15 10:46:28.219641 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.219648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b4b849546-cmq8n"]
Jul 15 10:46:28.219863 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.219748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.222262 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 15 10:46:28.222532 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 15 10:46:28.222634 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 15 10:46:28.222688 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 15 10:46:28.222767 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2ltv8\""
Jul 15 10:46:28.222864 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222844    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 15 10:46:28.222960 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.222890    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 15 10:46:28.229730 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.229711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-trusted-ca\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.229825 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.229757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-config\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.229825 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.229806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhwm\" (UniqueName: \"kubernetes.io/projected/acba8a5e-eb80-4b88-aa63-a33817400d0b-kube-api-access-hlhwm\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.229905 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.229835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acba8a5e-eb80-4b88-aa63-a33817400d0b-serving-cert\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.330755 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-config\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.330755 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.330988 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-stats-auth\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.330988 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhwm\" (UniqueName: \"kubernetes.io/projected/acba8a5e-eb80-4b88-aa63-a33817400d0b-kube-api-access-hlhwm\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.330988 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jn6x\" (UniqueName: \"kubernetes.io/projected/aea22a48-4abf-4d49-9058-99d9ad240187-kube-api-access-6jn6x\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.330988 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2228p\" (UniqueName: \"kubernetes.io/projected/bc11f264-bef2-4fd9-92ba-16f892918e19-kube-api-access-2228p\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.331175 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.330991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acba8a5e-eb80-4b88-aa63-a33817400d0b-serving-cert\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.331175 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.331175 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc11f264-bef2-4fd9-92ba-16f892918e19-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.331175 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-default-certificate\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.331315 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc11f264-bef2-4fd9-92ba-16f892918e19-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.331315 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-trusted-ca\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.331511 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.331491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-config\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.332026 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.332003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acba8a5e-eb80-4b88-aa63-a33817400d0b-trusted-ca\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.334457 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.334402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acba8a5e-eb80-4b88-aa63-a33817400d0b-serving-cert\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.337437 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.337411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:28.341235 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.341219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhwm\" (UniqueName: \"kubernetes.io/projected/acba8a5e-eb80-4b88-aa63-a33817400d0b-kube-api-access-hlhwm\") pod \"console-operator-7866497494-hc5rp\" (UID: \"acba8a5e-eb80-4b88-aa63-a33817400d0b\") " pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.415519 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.415474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:28.427730 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.427706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:28.431758 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.431732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jn6x\" (UniqueName: \"kubernetes.io/projected/aea22a48-4abf-4d49-9058-99d9ad240187-kube-api-access-6jn6x\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.431872 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.431772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2228p\" (UniqueName: \"kubernetes.io/projected/bc11f264-bef2-4fd9-92ba-16f892918e19-kube-api-access-2228p\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.431936 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.431916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.431987 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.431951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc11f264-bef2-4fd9-92ba-16f892918e19-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.432035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.431990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-default-certificate\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.432035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.432008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc11f264-bef2-4fd9-92ba-16f892918e19-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.432175 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.432056    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 15 10:46:28.432175 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.432136    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:28.932112885 +0000 UTC m=+54.766857329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : secret "router-metrics-certs-default" not found
Jul 15 10:46:28.432259 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.432233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.432304 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.432262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-stats-auth\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.432423 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.432406    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:28.932390901 +0000 UTC m=+54.767135343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : configmap references non-existent config key: service-ca.crt
Jul 15 10:46:28.432745 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.432685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc11f264-bef2-4fd9-92ba-16f892918e19-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.434399 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.434375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc11f264-bef2-4fd9-92ba-16f892918e19-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.434639 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.434621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-stats-auth\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.434867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.434844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-default-certificate\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.442766 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.442745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jn6x\" (UniqueName: \"kubernetes.io/projected/aea22a48-4abf-4d49-9058-99d9ad240187-kube-api-access-6jn6x\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.442891 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.442816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2228p\" (UniqueName: \"kubernetes.io/projected/bc11f264-bef2-4fd9-92ba-16f892918e19-kube-api-access-2228p\") pod \"kube-storage-version-migrator-operator-799cb7f958-wwxct\" (UID: \"bc11f264-bef2-4fd9-92ba-16f892918e19\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.505562 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.505533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"
Jul 15 10:46:28.568640 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.568610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-hc5rp"]
Jul 15 10:46:28.573624 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:28.573586    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacba8a5e_eb80_4b88_aa63_a33817400d0b.slice/crio-535ca906a94743a9d81a8d0e2a23f9b107f614a899d3d6083595014dcfb5dfa8 WatchSource:0}: Error finding container 535ca906a94743a9d81a8d0e2a23f9b107f614a899d3d6083595014dcfb5dfa8: Status 404 returned error can't find the container with id 535ca906a94743a9d81a8d0e2a23f9b107f614a899d3d6083595014dcfb5dfa8
Jul 15 10:46:28.652482 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.652451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct"]
Jul 15 10:46:28.656015 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:28.655990    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc11f264_bef2_4fd9_92ba_16f892918e19.slice/crio-82f99bf873019c19ac1f848d66cd509bf9536dd386d7564953e2116e54fd7575 WatchSource:0}: Error finding container 82f99bf873019c19ac1f848d66cd509bf9536dd386d7564953e2116e54fd7575: Status 404 returned error can't find the container with id 82f99bf873019c19ac1f848d66cd509bf9536dd386d7564953e2116e54fd7575
Jul 15 10:46:28.921428 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.921329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hc5rp" event={"ID":"acba8a5e-eb80-4b88-aa63-a33817400d0b","Type":"ContainerStarted","Data":"535ca906a94743a9d81a8d0e2a23f9b107f614a899d3d6083595014dcfb5dfa8"}
Jul 15 10:46:28.922253 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.922229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct" event={"ID":"bc11f264-bef2-4fd9-92ba-16f892918e19","Type":"ContainerStarted","Data":"82f99bf873019c19ac1f848d66cd509bf9536dd386d7564953e2116e54fd7575"}
Jul 15 10:46:28.943541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.943515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.943643 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:28.943601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:28.943686 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.943654    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 15 10:46:28.943719 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.943700    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:29.943684897 +0000 UTC m=+55.778429325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : secret "router-metrics-certs-default" not found
Jul 15 10:46:28.943719 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:28.943713    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:29.943706744 +0000 UTC m=+55.778451172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : configmap references non-existent config key: service-ca.crt
Jul 15 10:46:29.951974 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:29.951936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:29.952434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:29.951998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:29.952434 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:29.952120    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 15 10:46:29.952434 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:29.952142    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.952120541 +0000 UTC m=+57.786864980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : configmap references non-existent config key: service-ca.crt
Jul 15 10:46:29.952434 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:29.952164    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.95215331 +0000 UTC m=+57.786897739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : secret "router-metrics-certs-default" not found
Jul 15 10:46:30.664368 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:30.664316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:30.765139 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:30.765106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:31.932065 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.932006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct" event={"ID":"bc11f264-bef2-4fd9-92ba-16f892918e19","Type":"ContainerStarted","Data":"ac21299849b5112acbae3a66ddba58285f1d054f38354eb9242c33e0f6f65043"}
Jul 15 10:46:31.933699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.933680    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/0.log"
Jul 15 10:46:31.933784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.933715    2565 generic.go:356] "Generic (PLEG): container finished" podID="acba8a5e-eb80-4b88-aa63-a33817400d0b" containerID="d435990eb1a20a125abcdf16d0a408b8f99cb1341690f10e58b60278e52cb562" exitCode=255
Jul 15 10:46:31.933784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.933754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hc5rp" event={"ID":"acba8a5e-eb80-4b88-aa63-a33817400d0b","Type":"ContainerDied","Data":"d435990eb1a20a125abcdf16d0a408b8f99cb1341690f10e58b60278e52cb562"}
Jul 15 10:46:31.933944 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.933930    2565 scope.go:117] "RemoveContainer" containerID="d435990eb1a20a125abcdf16d0a408b8f99cb1341690f10e58b60278e52cb562"
Jul 15 10:46:31.950562 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.950518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct" podStartSLOduration=1.617001092 podStartE2EDuration="3.950499706s" totalImagesPullingTime="2.333498614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:31.950198922 +0000 UTC m=+57.784943396" watchObservedRunningTime="2026-07-15 10:46:31.950499706 +0000 UTC m=+57.785244157"
Jul 15 10:46:31.968919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.968893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:31.969076 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:31.969056    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.969017005 +0000 UTC m=+61.803761439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : configmap references non-existent config key: service-ca.crt
Jul 15 10:46:31.969568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:31.969156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:31.970182 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:31.969983    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 15 10:46:31.970182 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:31.970082    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.970033875 +0000 UTC m=+61.804778309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : secret "router-metrics-certs-default" not found
Jul 15 10:46:32.338903 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.338870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:32.438104 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.438058    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:32.528433 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.528402    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:32.712202 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.712112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:32.808177 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.808148    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:32.938151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:46:32.938525 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938511    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/0.log"
Jul 15 10:46:32.938571 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938545    2565 generic.go:356] "Generic (PLEG): container finished" podID="acba8a5e-eb80-4b88-aa63-a33817400d0b" containerID="7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c" exitCode=255
Jul 15 10:46:32.938609 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hc5rp" event={"ID":"acba8a5e-eb80-4b88-aa63-a33817400d0b","Type":"ContainerDied","Data":"7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c"}
Jul 15 10:46:32.938650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938639    2565 scope.go:117] "RemoveContainer" containerID="d435990eb1a20a125abcdf16d0a408b8f99cb1341690f10e58b60278e52cb562"
Jul 15 10:46:32.938851 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:32.938829    2565 scope.go:117] "RemoveContainer" containerID="7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c"
Jul 15 10:46:32.939081 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:32.939031    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hc5rp_openshift-console-operator(acba8a5e-eb80-4b88-aa63-a33817400d0b)\"" pod="openshift-console-operator/console-operator-7866497494-hc5rp" podUID="acba8a5e-eb80-4b88-aa63-a33817400d0b"
Jul 15 10:46:33.112017 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.111986    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:33.213231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.213203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:33.396645 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.396565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:33.495797 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.495764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:33.942395 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.942366    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:46:33.942846 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:33.942755    2565 scope.go:117] "RemoveContainer" containerID="7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c"
Jul 15 10:46:33.942952 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:33.942929    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hc5rp_openshift-console-operator(acba8a5e-eb80-4b88-aa63-a33817400d0b)\"" pod="openshift-console-operator/console-operator-7866497494-hc5rp" podUID="acba8a5e-eb80-4b88-aa63-a33817400d0b"
Jul 15 10:46:34.150212 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.150171    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdktl"]
Jul 15 10:46:34.182158 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.182122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:34.185127 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.185110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdktl"]
Jul 15 10:46:34.185231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.185214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.187498 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.187464    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 15 10:46:34.187625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.187532    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 15 10:46:34.187625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.187482    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 15 10:46:34.187819 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.187800    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 15 10:46:34.187915 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.187900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zgdp7\""
Jul 15 10:46:34.285405 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.285325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-data-volume\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.285405 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.285357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mx2\" (UniqueName: \"kubernetes.io/projected/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-api-access-s9mx2\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.285405 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.285381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.285589 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.285493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.285589 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.285521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-crio-socket\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.386777 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.386738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-data-volume\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.386777 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.386780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mx2\" (UniqueName: \"kubernetes.io/projected/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-api-access-s9mx2\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.387011 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.386812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.387011 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.386912    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.387011 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.386977    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:34.886957949 +0000 UTC m=+60.721702381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.387011 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.386996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.387247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.387028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-crio-socket\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.387247 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.387108    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.387247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.387160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-crio-socket\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.387247 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.387180    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:34.887167608 +0000 UTC m=+60.721912037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.387247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.387109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-data-volume\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.395784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.395763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mx2\" (UniqueName: \"kubernetes.io/projected/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-api-access-s9mx2\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.634584 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.634556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:34.736631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.736587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:34.875605 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.875578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg9qp"
Jul 15 10:46:34.890759 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.890691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.890877 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:34.890786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:34.890941 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.890893    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.891082 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.890893    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.891082 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.890965    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.8909511 +0000 UTC m=+61.725695527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.891082 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:34.891009    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.890991073 +0000 UTC m=+61.725735502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:35.079093 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.079063    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:35.244106 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.244000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:35.899058 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.899016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:35.899226 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.899109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:35.899226 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.899176    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:35.899320 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.899256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:37.899241227 +0000 UTC m=+63.733985657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:35.899320 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.899182    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:35.899320 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.899312    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:37.899301379 +0000 UTC m=+63.734045808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:35.999778 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.999747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:35.999930 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:35.999789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:35.999930 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.999882    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 15 10:46:35.999930 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.999907    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:43.999889314 +0000 UTC m=+69.834633741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : configmap references non-existent config key: service-ca.crt
Jul 15 10:46:35.999930 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:35.999931    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs podName:aea22a48-4abf-4d49-9058-99d9ad240187 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:43.999923137 +0000 UTC m=+69.834667566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs") pod "router-default-6b4b849546-cmq8n" (UID: "aea22a48-4abf-4d49-9058-99d9ad240187") : secret "router-metrics-certs-default" not found
Jul 15 10:46:36.045427 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:36.045396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:36.693249 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:36.693224    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ckfng_09c549f7-a074-4a6a-a103-6517d81ce013/dns-node-resolver/0.log"
Jul 15 10:46:36.768752 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:36.768717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:36.875158 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:36.875125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.033930 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.033847    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.134564 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.134529    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.305211 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.305133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.391351 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.391318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.488199 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.488170    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2h2kf_5a5645dd-b4fd-4aff-a198-f743157e501a/node-ca/0.log"
Jul 15 10:46:37.495865 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.495830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:37.914356 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.914325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:37.914803 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:37.914399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:37.914803 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:37.914474    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:37.914803 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:37.914495    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:37.914803 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:37.914546    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:41.914531934 +0000 UTC m=+67.749276363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:37.914803 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:37.914559    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:41.914553554 +0000 UTC m=+67.749297981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:38.415769 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.415739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:38.415926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.415793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:38.416144 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.416132    2565 scope.go:117] "RemoveContainer" containerID="7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c"
Jul 15 10:46:38.416294 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:38.416276    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-hc5rp_openshift-console-operator(acba8a5e-eb80-4b88-aa63-a33817400d0b)\"" pod="openshift-console-operator/console-operator-7866497494-hc5rp" podUID="acba8a5e-eb80-4b88-aa63-a33817400d0b"
Jul 15 10:46:38.445946 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.445919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:38.687422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.687350    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rztx6_b55bedd1-7fc7-4595-a8fa-7b5801ae43ee/serve-healthcheck-canary/0.log"
Jul 15 10:46:38.847187 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:38.847152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:39.043968 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:39.043896    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:39.258593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:39.258559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:39.288357 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:39.288325    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-wwxct_bc11f264-bef2-4fd9-92ba-16f892918e19/kube-storage-version-migrator-operator/0.log"
Jul 15 10:46:39.484006 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:39.483974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:40.434856 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.434823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:46:40.434856 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.434858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:40.435276 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.434927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:40.438476 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.438456    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:46:40.438770 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.438754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:46:40.439339 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.439324    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:46:40.446079 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:40.446064    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 15 10:46:40.446135 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:40.446121    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs podName:810a3b89-0a7c-48b6-a769-d93e1d61fe48 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:44.446103649 +0000 UTC m=+130.280848077 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs") pod "network-metrics-daemon-kkfx8" (UID: "810a3b89-0a7c-48b6-a769-d93e1d61fe48") : secret "metrics-daemon-secret" not found
Jul 15 10:46:40.448257 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.448231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24527bdc-899a-498d-ae7c-8c09a08e921d-original-pull-secret\") pod \"global-pull-secret-syncer-lctcq\" (UID: \"24527bdc-899a-498d-ae7c-8c09a08e921d\") " pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:40.448623 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.448608    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:46:40.458695 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.458676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ksj\" (UniqueName: \"kubernetes.io/projected/96cf6719-f021-43f0-9eeb-d17c1f893325-kube-api-access-s4ksj\") pod \"network-check-target-mp8nx\" (UID: \"96cf6719-f021-43f0-9eeb-d17c1f893325\") " pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:40.691055 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.690956    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lctcq"
Jul 15 10:46:40.700093 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.700068    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhf7j\""
Jul 15 10:46:40.708415 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.708393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:40.828542 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.828515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lctcq"]
Jul 15 10:46:40.833011 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:40.832986    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24527bdc_899a_498d_ae7c_8c09a08e921d.slice/crio-cee4125a8cedf80a9c867108ab876d7ab8d6006b15468b52583546b89deba4eb WatchSource:0}: Error finding container cee4125a8cedf80a9c867108ab876d7ab8d6006b15468b52583546b89deba4eb: Status 404 returned error can't find the container with id cee4125a8cedf80a9c867108ab876d7ab8d6006b15468b52583546b89deba4eb
Jul 15 10:46:40.848241 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.848192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp8nx"]
Jul 15 10:46:40.853225 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:40.853205    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cf6719_f021_43f0_9eeb_d17c1f893325.slice/crio-cfd105857c9ffa4b725170152a9c760e22895d3a75d031c040988562afa91e8d WatchSource:0}: Error finding container cfd105857c9ffa4b725170152a9c760e22895d3a75d031c040988562afa91e8d: Status 404 returned error can't find the container with id cfd105857c9ffa4b725170152a9c760e22895d3a75d031c040988562afa91e8d
Jul 15 10:46:40.917472 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.917436    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:40.961546 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.961475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp8nx" event={"ID":"96cf6719-f021-43f0-9eeb-d17c1f893325","Type":"ContainerStarted","Data":"cfd105857c9ffa4b725170152a9c760e22895d3a75d031c040988562afa91e8d"}
Jul 15 10:46:40.962526 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:40.962502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lctcq" event={"ID":"24527bdc-899a-498d-ae7c-8c09a08e921d","Type":"ContainerStarted","Data":"cee4125a8cedf80a9c867108ab876d7ab8d6006b15468b52583546b89deba4eb"}
Jul 15 10:46:41.121371 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.121331    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:41.276462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.276374    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:41.412958 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.412919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:41.443009 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.442979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:41.445876 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.445848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c21dea71-cff4-4203-95e7-b1e5112fa9d2-metrics-tls\") pod \"dns-default-xv7x9\" (UID: \"c21dea71-cff4-4203-95e7-b1e5112fa9d2\") " pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:41.498600 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.498562    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:41.600972 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.600909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:41.729408 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.729375    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8cmqw\""
Jul 15 10:46:41.737629 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.737601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:41.900555 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.900474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv7x9"]
Jul 15 10:46:41.904528 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:41.904503    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21dea71_cff4_4203_95e7_b1e5112fa9d2.slice/crio-cb4d15ba63bb84282a8c6b9a9635a14a3f64a8b712cbc70ad9e7792d508671f1 WatchSource:0}: Error finding container cb4d15ba63bb84282a8c6b9a9635a14a3f64a8b712cbc70ad9e7792d508671f1: Status 404 returned error can't find the container with id cb4d15ba63bb84282a8c6b9a9635a14a3f64a8b712cbc70ad9e7792d508671f1
Jul 15 10:46:41.947489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.947461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:41.947622 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.947526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:41.947622 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:41.947604    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:41.947743 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:41.947614    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:41.947743 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:41.947676    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:49.947656798 +0000 UTC m=+75.782401229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:41.947743 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:41.947693    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls podName:45a962f4-acc0-4a4a-87cf-f3cd6603ddf8 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:49.947685223 +0000 UTC m=+75.782429653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdktl" (UID: "45a962f4-acc0-4a4a-87cf-f3cd6603ddf8") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:41.966458 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:41.966399    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv7x9" event={"ID":"c21dea71-cff4-4203-95e7-b1e5112fa9d2","Type":"ContainerStarted","Data":"cb4d15ba63bb84282a8c6b9a9635a14a3f64a8b712cbc70ad9e7792d508671f1"}
Jul 15 10:46:42.597324 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:42.597285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:42.698596 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:42.698292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:42.847780 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:42.847700    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:43.039034 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:43.038998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:43.247711 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:43.247526    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:43.441901 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:43.441863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:44.066864 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.066830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:44.067337 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.066894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:44.068093 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.068063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea22a48-4abf-4d49-9058-99d9ad240187-service-ca-bundle\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:44.069953 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.069927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea22a48-4abf-4d49-9058-99d9ad240187-metrics-certs\") pod \"router-default-6b4b849546-cmq8n\" (UID: \"aea22a48-4abf-4d49-9058-99d9ad240187\") " pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:44.132188 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.132156    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2ltv8\""
Jul 15 10:46:44.138625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.138609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:44.510744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.510709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:44.848846 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:44.848798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.050468 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:45.050430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.245265 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:45.245154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.457087 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:45.457006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:45.531692 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:45.531622    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.172567 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.172540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b4b849546-cmq8n"]
Jul 15 10:46:46.176143 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:46.176122    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea22a48_4abf_4d49_9058_99d9ad240187.slice/crio-a9127466d2c34158dc0c2a967e57210592a1b757838e0ebc73313c0ffbd02feb WatchSource:0}: Error finding container a9127466d2c34158dc0c2a967e57210592a1b757838e0ebc73313c0ffbd02feb: Status 404 returned error can't find the container with id a9127466d2c34158dc0c2a967e57210592a1b757838e0ebc73313c0ffbd02feb
Jul 15 10:46:46.250504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.250117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.480600 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.480565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.684963 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.684929    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.782676 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.782596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.879657 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.879622    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:46.985989 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.985947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp8nx" event={"ID":"96cf6719-f021-43f0-9eeb-d17c1f893325","Type":"ContainerStarted","Data":"fe5c27c52727639f3fd2f944fc3c4087d54a203884c3c3f2b75748b45ebc2763"}
Jul 15 10:46:46.986204 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.986083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:46:46.987682 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.987662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv7x9" event={"ID":"c21dea71-cff4-4203-95e7-b1e5112fa9d2","Type":"ContainerStarted","Data":"1b53437a666e06de103214badc37cb457e1d240bf5d87bfff810b7f2937e9da1"}
Jul 15 10:46:46.987813 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.987699    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:46.987813 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.987716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv7x9" event={"ID":"c21dea71-cff4-4203-95e7-b1e5112fa9d2","Type":"ContainerStarted","Data":"45cfcab5e4137ece0ca78a3b67d3c04e57c59a52ba9b8a09ee6e04505067afa5"}
Jul 15 10:46:46.989152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.989130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lctcq" event={"ID":"24527bdc-899a-498d-ae7c-8c09a08e921d","Type":"ContainerStarted","Data":"99a993d26eeb4ecc5bb02337d383a44d7ab9b2a11667799beb4c0e1f41aa5d2b"}
Jul 15 10:46:46.990505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.990486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b4b849546-cmq8n" event={"ID":"aea22a48-4abf-4d49-9058-99d9ad240187","Type":"ContainerStarted","Data":"6a54d6dc47c49a7b675772a8b0c411ae35d81f65fd366bac104646463720c8e8"}
Jul 15 10:46:46.990592 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:46.990521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b4b849546-cmq8n" event={"ID":"aea22a48-4abf-4d49-9058-99d9ad240187","Type":"ContainerStarted","Data":"a9127466d2c34158dc0c2a967e57210592a1b757838e0ebc73313c0ffbd02feb"}
Jul 15 10:46:47.004196 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.004153    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mp8nx" podStartSLOduration=66.833471534 podStartE2EDuration="1m12.004139542s" totalImagesPullingTime="5.170668008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.003469578 +0000 UTC m=+72.838214059" watchObservedRunningTime="2026-07-15 10:46:47.004139542 +0000 UTC m=+72.838883991"
Jul 15 10:46:47.027936 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.027888    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xv7x9" podStartSLOduration=33.874514271 podStartE2EDuration="38.027871835s" totalImagesPullingTime="4.153357564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.027136448 +0000 UTC m=+72.861880878" watchObservedRunningTime="2026-07-15 10:46:47.027871835 +0000 UTC m=+72.862616286"
Jul 15 10:46:47.086478 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.086431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b4b849546-cmq8n" podStartSLOduration=19.086416094 podStartE2EDuration="19.086416094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.057864284 +0000 UTC m=+72.892608754" watchObservedRunningTime="2026-07-15 10:46:47.086416094 +0000 UTC m=+72.921160539"
Jul 15 10:46:47.138767 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.138737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:47.141534 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.141513    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:47.173015 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.172968    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lctcq" podStartSLOduration=66.955720229 podStartE2EDuration="1m12.17295365s" totalImagesPullingTime="5.217233421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.087470202 +0000 UTC m=+72.922214651" watchObservedRunningTime="2026-07-15 10:46:47.17295365 +0000 UTC m=+73.007698094"
Jul 15 10:46:47.629266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.629237    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:47.697228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.697196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:47.844968 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.844935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:47.995325 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.995234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:47.996230 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:47.996210    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b4b849546-cmq8n"
Jul 15 10:46:49.668112 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:49.668083    2565 scope.go:117] "RemoveContainer" containerID="7b0a6884fc20ef25817864f7704f2b97abf590a8c828c4bf29d79e5531e1100c"
Jul 15 10:46:50.001396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.001321    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:46:50.001396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.001373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-hc5rp" event={"ID":"acba8a5e-eb80-4b88-aa63-a33817400d0b","Type":"ContainerStarted","Data":"c50b51b445c1e087064518e6ec2aca237720a4baeb13b330bcf1d777872882b0"}
Jul 15 10:46:50.001762 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.001744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:50.006225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.006202    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-hc5rp"
Jul 15 10:46:50.017922 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.017897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:50.018061 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.017956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:50.018407 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.018388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:50.020505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.020482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45a962f4-acc0-4a4a-87cf-f3cd6603ddf8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdktl\" (UID: \"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8\") " pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:50.020972 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.020938    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-hc5rp" podStartSLOduration=19.607802205 podStartE2EDuration="22.020922849s" totalImagesPullingTime="2.413120644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:50.019751288 +0000 UTC m=+75.854495758" watchObservedRunningTime="2026-07-15 10:46:50.020922849 +0000 UTC m=+75.855667299"
Jul 15 10:46:50.096163 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.096125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:50.096303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.096212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zgdp7\""
Jul 15 10:46:50.104266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.104246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdktl"
Jul 15 10:46:50.181081 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.181054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:50.239627 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.239596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdktl"]
Jul 15 10:46:50.245085 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:50.245038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a962f4_acc0_4a4a_87cf_f3cd6603ddf8.slice/crio-9f52c302ec98f746824fee302727a76d5ecd6084c138a98a89f7dc02ee7f0f3d WatchSource:0}: Error finding container 9f52c302ec98f746824fee302727a76d5ecd6084c138a98a89f7dc02ee7f0f3d: Status 404 returned error can't find the container with id 9f52c302ec98f746824fee302727a76d5ecd6084c138a98a89f7dc02ee7f0f3d
Jul 15 10:46:50.250983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.250956    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:50.447995 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.447959    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:50.786406 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.786286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:50.883193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:50.883120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:51.008155 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:51.008108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdktl" event={"ID":"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8","Type":"ContainerStarted","Data":"9f52c302ec98f746824fee302727a76d5ecd6084c138a98a89f7dc02ee7f0f3d"}
Jul 15 10:46:51.188075 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:51.188008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:51.258067 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:51.258020    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:51.805759 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:51.805727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:51.986571 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:51.986533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:52.242148 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:52.242106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:53.203561 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:53.203525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:53.270138 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:53.270105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:56.030266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:56.030169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdktl" event={"ID":"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8","Type":"ContainerStarted","Data":"f0217451a8b5f9ccb80fe6d77ca4b50936d1d08de18284fe9336f3ee3570a0e5"}
Jul 15 10:46:56.997791 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:56.997761    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xv7x9"
Jul 15 10:46:57.116799 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.116756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"]
Jul 15 10:46:57.124463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.124439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.133309 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.133256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 15 10:46:57.142938 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.142911    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 15 10:46:57.143135 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.143108    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 15 10:46:57.143262 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.143237    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 15 10:46:57.170235 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.170203    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"]
Jul 15 10:46:57.186476 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.186448    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"]
Jul 15 10:46:57.189150 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.189128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:46:57.193524 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.193503    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-726xw\""
Jul 15 10:46:57.193770 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.193754    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 15 10:46:57.205441 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.205420    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"]
Jul 15 10:46:57.207432 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.207410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"
Jul 15 10:46:57.209315 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.209293    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-578fc5d65-mrxjh"]
Jul 15 10:46:57.209886 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.209864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gn6cc\""
Jul 15 10:46:57.211766 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.211746    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"]
Jul 15 10:46:57.211863 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.211847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.216007 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.215953    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-mqg6r\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqg6r\"" type="*v1.Secret"
Jul 15 10:46:57.216297 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.216270    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Jul 15 10:46:57.216625 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.216599    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret"
Jul 15 10:46:57.216916 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.216871    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Jul 15 10:46:57.217747 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.217718    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Jul 15 10:46:57.229659 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.229630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"]
Jul 15 10:46:57.247354 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.247325    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-578fc5d65-mrxjh"]
Jul 15 10:46:57.274936 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.274863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.274936 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.274909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820df008-db97-4b72-b553-4dc34bd8fa3d-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-564nt\" (UID: \"820df008-db97-4b72-b553-4dc34bd8fa3d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:46:57.275137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.274967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llf6g\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.275411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tct\" (UniqueName: \"kubernetes.io/projected/b0a4ae55-0df3-44a0-9c65-182cd3fd8eff-kube-api-access-49tct\") pod \"network-check-source-869dcfcdcd-gszj8\" (UID: \"b0a4ae55-0df3-44a0-9c65-182cd3fd8eff\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"
Jul 15 10:46:57.275411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7b02c9-087a-4060-b6ef-3b227ab23108-tmp\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.275411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce7b02c9-087a-4060-b6ef-3b227ab23108-klusterlet-config\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.275411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.275339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzgj\" (UniqueName: \"kubernetes.io/projected/ce7b02c9-087a-4060-b6ef-3b227ab23108-kube-api-access-sdzgj\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.296396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.296369    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-578fc5d65-mrxjh"]
Jul 15 10:46:57.296629 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:57.296597    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-llf6g registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-578fc5d65-mrxjh" podUID="07a858a4-8657-4fb8-a934-5f045a0c69d4"
Jul 15 10:46:57.337191 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.337148    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:57.376367 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820df008-db97-4b72-b553-4dc34bd8fa3d-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-564nt\" (UID: \"820df008-db97-4b72-b553-4dc34bd8fa3d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:46:57.376541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llf6g\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tct\" (UniqueName: \"kubernetes.io/projected/b0a4ae55-0df3-44a0-9c65-182cd3fd8eff-kube-api-access-49tct\") pod \"network-check-source-869dcfcdcd-gszj8\" (UID: \"b0a4ae55-0df3-44a0-9c65-182cd3fd8eff\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7b02c9-087a-4060-b6ef-3b227ab23108-tmp\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.376776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce7b02c9-087a-4060-b6ef-3b227ab23108-klusterlet-config\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.377145 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzgj\" (UniqueName: \"kubernetes.io/projected/ce7b02c9-087a-4060-b6ef-3b227ab23108-kube-api-access-sdzgj\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.377145 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.376758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.377247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.377225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce7b02c9-087a-4060-b6ef-3b227ab23108-tmp\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.377509 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.377443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.379356 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.379334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820df008-db97-4b72-b553-4dc34bd8fa3d-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-564nt\" (UID: \"820df008-db97-4b72-b553-4dc34bd8fa3d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:46:57.379781 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.379762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce7b02c9-087a-4060-b6ef-3b227ab23108-klusterlet-config\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.390495 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.390455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llf6g\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.390993 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.390973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzgj\" (UniqueName: \"kubernetes.io/projected/ce7b02c9-087a-4060-b6ef-3b227ab23108-kube-api-access-sdzgj\") pod \"klusterlet-addon-workmgr-575596866b-cwv44\" (UID: \"ce7b02c9-087a-4060-b6ef-3b227ab23108\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.391366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.391344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:57.391625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.391602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tct\" (UniqueName: \"kubernetes.io/projected/b0a4ae55-0df3-44a0-9c65-182cd3fd8eff-kube-api-access-49tct\") pod \"network-check-source-869dcfcdcd-gszj8\" (UID: \"b0a4ae55-0df3-44a0-9c65-182cd3fd8eff\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"
Jul 15 10:46:57.430163 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.430127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:46:57.436945 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.436918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:46:57.500210 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.500175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:46:57.521007 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:57.520969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"
Jul 15 10:46:58.032667 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.032622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 15 10:46:58.037383 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.037351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.037383 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.037370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.064194 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.064163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.174583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.174555    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqg6r\""
Jul 15 10:46:58.182821 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.182793    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.182967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.182832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.182967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.182882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.182967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.182906    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.182967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.182925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llf6g\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.183422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.183173    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:46:58.183422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.183384    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:46:58.183422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.183393    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:46:58.185284 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.185257    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:46:58.185409 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.185392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g" (OuterVolumeSpecName: "kube-api-access-llf6g") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "kube-api-access-llf6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:46:58.283704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.283631    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llf6g\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-kube-api-access-llf6g\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:58.283704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.283661    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-bound-sa-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:58.283704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.283670    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07a858a4-8657-4fb8-a934-5f045a0c69d4-ca-trust-extracted\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:58.283704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.283679    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-trusted-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:58.283704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.283694    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-certificates\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:58.377529 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377495    2565 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377695 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377497    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377695 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377612    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578fc5d65-mrxjh: failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377695 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377628    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets podName:07a858a4-8657-4fb8-a934-5f045a0c69d4 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:58.877606064 +0000 UTC m=+84.712350523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets") pod "image-registry-578fc5d65-mrxjh" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4") : failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377695 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377495    2565 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377878 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377675    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls podName:07a858a4-8657-4fb8-a934-5f045a0c69d4 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:58.877655534 +0000 UTC m=+84.712399965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls") pod "image-registry-578fc5d65-mrxjh" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4") : failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.377878 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:46:58.377728    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration podName:07a858a4-8657-4fb8-a934-5f045a0c69d4 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:58.877714569 +0000 UTC m=+84.712458996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration") pod "image-registry-578fc5d65-mrxjh" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4") : failed to sync secret cache: timed out waiting for the condition
Jul 15 10:46:58.469700 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.469663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 15 10:46:58.497122 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.497093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 15 10:46:58.588449 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.588418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 15 10:46:58.888630 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.888592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.888794 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.888648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.888794 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.888687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.891893 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.891745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.891893 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.891845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.892858 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.892829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") pod \"image-registry-578fc5d65-mrxjh\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") " pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:58.961470 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.961436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"]
Jul 15 10:46:58.965196 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:58.965168    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820df008_db97_4b72_b553_4dc34bd8fa3d.slice/crio-513868fa5c0f3b046bcb9a38530c94adb1ee541840cf1cb200ba5b5ab7d116cc WatchSource:0}: Error finding container 513868fa5c0f3b046bcb9a38530c94adb1ee541840cf1cb200ba5b5ab7d116cc: Status 404 returned error can't find the container with id 513868fa5c0f3b046bcb9a38530c94adb1ee541840cf1cb200ba5b5ab7d116cc
Jul 15 10:46:58.970505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.970478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"]
Jul 15 10:46:58.973399 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:58.973375    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7b02c9_087a_4060_b6ef_3b227ab23108.slice/crio-1f4749d63811fb16d5c57bf4387989c88205e2a250750e4e5f311004879acc9d WatchSource:0}: Error finding container 1f4749d63811fb16d5c57bf4387989c88205e2a250750e4e5f311004879acc9d: Status 404 returned error can't find the container with id 1f4749d63811fb16d5c57bf4387989c88205e2a250750e4e5f311004879acc9d
Jul 15 10:46:58.989276 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.989253    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.989363 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.989286    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.989363 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.989316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") pod \"07a858a4-8657-4fb8-a934-5f045a0c69d4\" (UID: \"07a858a4-8657-4fb8-a934-5f045a0c69d4\") "
Jul 15 10:46:58.991483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.991441    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:46:58.991483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.991469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:46:58.991483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.991439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "07a858a4-8657-4fb8-a934-5f045a0c69d4" (UID: "07a858a4-8657-4fb8-a934-5f045a0c69d4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:46:58.997167 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:58.997145    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8"]
Jul 15 10:46:59.000036 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:46:59.000015    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a4ae55_0df3_44a0_9c65_182cd3fd8eff.slice/crio-189e4baef3fd63d34a54aeae5c142eb848fbc12e3855d88c7af3cbd288d83b34 WatchSource:0}: Error finding container 189e4baef3fd63d34a54aeae5c142eb848fbc12e3855d88c7af3cbd288d83b34: Status 404 returned error can't find the container with id 189e4baef3fd63d34a54aeae5c142eb848fbc12e3855d88c7af3cbd288d83b34
Jul 15 10:46:59.042845 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.042813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44" event={"ID":"ce7b02c9-087a-4060-b6ef-3b227ab23108","Type":"ContainerStarted","Data":"1f4749d63811fb16d5c57bf4387989c88205e2a250750e4e5f311004879acc9d"}
Jul 15 10:46:59.044339 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.044310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt" event={"ID":"820df008-db97-4b72-b553-4dc34bd8fa3d","Type":"ContainerStarted","Data":"513868fa5c0f3b046bcb9a38530c94adb1ee541840cf1cb200ba5b5ab7d116cc"}
Jul 15 10:46:59.046433 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.046407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdktl" event={"ID":"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8","Type":"ContainerStarted","Data":"f4d6fb331f01f6237456a969773ea972e0524212919a9627f1dc97fc7068ad01"}
Jul 15 10:46:59.047462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.047442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578fc5d65-mrxjh"
Jul 15 10:46:59.047550 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.047463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8" event={"ID":"b0a4ae55-0df3-44a0-9c65-182cd3fd8eff","Type":"ContainerStarted","Data":"189e4baef3fd63d34a54aeae5c142eb848fbc12e3855d88c7af3cbd288d83b34"}
Jul 15 10:46:59.090438 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.090409    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07a858a4-8657-4fb8-a934-5f045a0c69d4-registry-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:59.090438 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.090436    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-image-registry-private-configuration\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:59.090438 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.090445    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07a858a4-8657-4fb8-a934-5f045a0c69d4-installation-pull-secrets\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:46:59.096350 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.096270    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-578fc5d65-mrxjh"]
Jul 15 10:46:59.103271 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:46:59.103242    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-578fc5d65-mrxjh"]
Jul 15 10:47:00.053963 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:00.053552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8" event={"ID":"b0a4ae55-0df3-44a0-9c65-182cd3fd8eff","Type":"ContainerStarted","Data":"ceac68682b4d0a9d6e2b47266e17ec253d9b47d190fdc1e1e024ec0e87aabe5e"}
Jul 15 10:47:00.069935 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:00.069877    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gszj8" podStartSLOduration=3.0698532 podStartE2EDuration="3.0698532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:00.069427125 +0000 UTC m=+85.904171591" watchObservedRunningTime="2026-07-15 10:47:00.0698532 +0000 UTC m=+85.904597651"
Jul 15 10:47:00.681746 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:00.681709    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a858a4-8657-4fb8-a934-5f045a0c69d4" path="/var/lib/kubelet/pods/07a858a4-8657-4fb8-a934-5f045a0c69d4/volumes"
Jul 15 10:47:00.777281 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:00.777218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:00.878256 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:00.878216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:01.060399 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.060066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt" event={"ID":"820df008-db97-4b72-b553-4dc34bd8fa3d","Type":"ContainerStarted","Data":"f6688c11d634ed2103064ad8fe49ea3f0cb3fee06128f4253fa6289fba77ab57"}
Jul 15 10:47:01.060790 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.060453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:47:01.067573 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.067526    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt"
Jul 15 10:47:01.081430 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.080692    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-564nt" podStartSLOduration=2.818322535 podStartE2EDuration="4.080671514s" totalImagesPullingTime="1.262348979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:01.079274789 +0000 UTC m=+86.914019240" watchObservedRunningTime="2026-07-15 10:47:01.080671514 +0000 UTC m=+86.915415947"
Jul 15 10:47:01.131622 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.131535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:01.246498 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.246437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:01.347516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.347462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:01.884208 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.884155    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:01.935431 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.935400    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kpvpd"]
Jul 15 10:47:01.940225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.940199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:01.942842 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.942820    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 15 10:47:01.944106 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.943538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 15 10:47:01.944106 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.943778    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lhmq9\""
Jul 15 10:47:01.944106 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.943988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 15 10:47:01.945200 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.944733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 15 10:47:01.945200 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.944950    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 15 10:47:01.948784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.948715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kpvpd"]
Jul 15 10:47:01.967507 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:01.967476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:02.023332 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.023298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.023500 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.023361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf686\" (UniqueName: \"kubernetes.io/projected/2bd74670-f40a-4e73-b8f5-9ace70d532ef-kube-api-access-kf686\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.023500 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.023419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd74670-f40a-4e73-b8f5-9ace70d532ef-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.023500 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.023442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.060687 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.060626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:02.124489 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.124438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.124684 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.124524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf686\" (UniqueName: \"kubernetes.io/projected/2bd74670-f40a-4e73-b8f5-9ace70d532ef-kube-api-access-kf686\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.124684 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.124563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd74670-f40a-4e73-b8f5-9ace70d532ef-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.124684 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.124594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.125431 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.125405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd74670-f40a-4e73-b8f5-9ace70d532ef-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.127243 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.127221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.127365 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.127346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd74670-f40a-4e73-b8f5-9ace70d532ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.134512 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.134439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf686\" (UniqueName: \"kubernetes.io/projected/2bd74670-f40a-4e73-b8f5-9ace70d532ef-kube-api-access-kf686\") pod \"prometheus-operator-64785b6675-kpvpd\" (UID: \"2bd74670-f40a-4e73-b8f5-9ace70d532ef\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:02.251485 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:02.251449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd"
Jul 15 10:47:04.956669 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:04.956638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kpvpd"]
Jul 15 10:47:04.959569 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:04.959535    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd74670_f40a_4e73_b8f5_9ace70d532ef.slice/crio-9a001c9f7aec1aa9542f54c50823cf085a45390d9d58f525a1b6f479d4f80d1e WatchSource:0}: Error finding container 9a001c9f7aec1aa9542f54c50823cf085a45390d9d58f525a1b6f479d4f80d1e: Status 404 returned error can't find the container with id 9a001c9f7aec1aa9542f54c50823cf085a45390d9d58f525a1b6f479d4f80d1e
Jul 15 10:47:05.076681 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.076637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44" event={"ID":"ce7b02c9-087a-4060-b6ef-3b227ab23108","Type":"ContainerStarted","Data":"3f67326e81b7b973eb9d3b157b5aa968a8431e31103f16fded204c43a69c8f8e"}
Jul 15 10:47:05.076865 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.076837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:47:05.078838 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.078808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44"
Jul 15 10:47:05.079014 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.078987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdktl" event={"ID":"45a962f4-acc0-4a4a-87cf-f3cd6603ddf8","Type":"ContainerStarted","Data":"151f8b3c739a8e2c100d3acf011c6113161545c5939bd16564074657745867d3"}
Jul 15 10:47:05.080093 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.080074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd" event={"ID":"2bd74670-f40a-4e73-b8f5-9ace70d532ef","Type":"ContainerStarted","Data":"9a001c9f7aec1aa9542f54c50823cf085a45390d9d58f525a1b6f479d4f80d1e"}
Jul 15 10:47:05.100513 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.100464    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-575596866b-cwv44" podStartSLOduration=2.252851552 podStartE2EDuration="8.10044898s" totalImagesPullingTime="5.847597428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:05.100247429 +0000 UTC m=+90.934991879" watchObservedRunningTime="2026-07-15 10:47:05.10044898 +0000 UTC m=+90.935193430"
Jul 15 10:47:05.129111 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.129021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zdktl" podStartSLOduration=16.692241467 podStartE2EDuration="31.129003156s" totalImagesPullingTime="14.436761689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:05.128280788 +0000 UTC m=+90.963025238" watchObservedRunningTime="2026-07-15 10:47:05.129003156 +0000 UTC m=+90.963747700"
Jul 15 10:47:05.661095 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.661061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:05.758461 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:05.758429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:06.175967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:06.175935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:06.275978 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:06.275949    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:07.436419 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:07.436316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:08.093035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:08.092999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd" event={"ID":"2bd74670-f40a-4e73-b8f5-9ace70d532ef","Type":"ContainerStarted","Data":"564c4ec4ea836a878eb529af41111a51eaf59c621ae2621d48fcd17c2a609241"}
Jul 15 10:47:08.093212 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:08.093102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd" event={"ID":"2bd74670-f40a-4e73-b8f5-9ace70d532ef","Type":"ContainerStarted","Data":"de2afbce2344a9f2513f2ba532ce5ceeee67e9c9b75c2e807488d0b0d038d1f5"}
Jul 15 10:47:08.113844 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:08.113795    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-kpvpd" podStartSLOduration=4.905476376 podStartE2EDuration="7.11377952s" totalImagesPullingTime="2.208303144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:08.113345004 +0000 UTC m=+93.948089454" watchObservedRunningTime="2026-07-15 10:47:08.11377952 +0000 UTC m=+93.948523969"
Jul 15 10:47:09.234939 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:09.234905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:09.335565 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:09.335532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.084916 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.084880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.420442 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.420335    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.474570 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.474535    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"]
Jul 15 10:47:10.477311 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.477295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.480109 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.480088    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 15 10:47:10.480221 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.480126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g2rws\""
Jul 15 10:47:10.480297 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.480238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 15 10:47:10.483856 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.483838    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 15 10:47:10.495060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.495014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"]
Jul 15 10:47:10.556485 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.556456    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2nr7q"]
Jul 15 10:47:10.558801 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.558784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.562060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.562016    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 15 10:47:10.562271 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.562259    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 15 10:47:10.562701 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.562683    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 15 10:47:10.562791 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.562687    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q7hjh\""
Jul 15 10:47:10.588665 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.588852 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.588852 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.588852 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.588852 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.589078 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.588870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jmh\" (UniqueName: \"kubernetes.io/projected/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-api-access-q8jmh\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.673662 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.673579    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.689561 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.689695 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jmh\" (UniqueName: \"kubernetes.io/projected/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-api-access-q8jmh\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.689695 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-sys\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689695 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-tls\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689811 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689811 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-textfile\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689811 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-root\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689811 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-metrics-client-ca\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldm7\" (UniqueName: \"kubernetes.io/projected/ce9cab10-b113-42f4-9bf7-461420db4623-kube-api-access-xldm7\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-wtmp\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.690202 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.689994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.690202 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.690033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.690345 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.690319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.690400 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.690385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.690664 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.690604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.692473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.692446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.692558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.692492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.704936 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.704911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jmh\" (UniqueName: \"kubernetes.io/projected/7f3a4ff0-1cd9-489a-9c26-1b68750d76bd-kube-api-access-q8jmh\") pod \"kube-state-metrics-785778ff74-dpnhr\" (UID: \"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.778973 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.778936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.786789 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.786760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"
Jul 15 10:47:10.790867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-sys\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.790984 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-tls\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.790984 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.790984 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-textfile\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.790984 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790916    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-sys\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.790984 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.790962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-root\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-root\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-metrics-client-ca\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldm7\" (UniqueName: \"kubernetes.io/projected/ce9cab10-b113-42f4-9bf7-461420db4623-kube-api-access-xldm7\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-wtmp\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791513 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-textfile\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791513 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-wtmp\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791672 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.791733 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.791714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce9cab10-b113-42f4-9bf7-461420db4623-metrics-client-ca\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.793761 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.793736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-tls\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.793947 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.793926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce9cab10-b113-42f4-9bf7-461420db4623-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.800666 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.800645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldm7\" (UniqueName: \"kubernetes.io/projected/ce9cab10-b113-42f4-9bf7-461420db4623-kube-api-access-xldm7\") pod \"node-exporter-2nr7q\" (UID: \"ce9cab10-b113-42f4-9bf7-461420db4623\") " pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.868480 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.868453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nr7q"
Jul 15 10:47:10.872493 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.872464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:10.880501 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:10.880478    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9cab10_b113_42f4_9bf7_461420db4623.slice/crio-1e49a6e6817c80809222b0fe9f8557e2e86f1ef1c4a1dffce4ca5db33f5b9fc1 WatchSource:0}: Error finding container 1e49a6e6817c80809222b0fe9f8557e2e86f1ef1c4a1dffce4ca5db33f5b9fc1: Status 404 returned error can't find the container with id 1e49a6e6817c80809222b0fe9f8557e2e86f1ef1c4a1dffce4ca5db33f5b9fc1
Jul 15 10:47:10.924423 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:10.924355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dpnhr"]
Jul 15 10:47:10.927384 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:10.927340    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3a4ff0_1cd9_489a_9c26_1b68750d76bd.slice/crio-8b4e021a95945c12df668af2c7e463d9788e10a2061fd8d719b975fa785ab936 WatchSource:0}: Error finding container 8b4e021a95945c12df668af2c7e463d9788e10a2061fd8d719b975fa785ab936: Status 404 returned error can't find the container with id 8b4e021a95945c12df668af2c7e463d9788e10a2061fd8d719b975fa785ab936
Jul 15 10:47:11.105366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.105327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr" event={"ID":"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd","Type":"ContainerStarted","Data":"8b4e021a95945c12df668af2c7e463d9788e10a2061fd8d719b975fa785ab936"}
Jul 15 10:47:11.106390 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.106366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nr7q" event={"ID":"ce9cab10-b113-42f4-9bf7-461420db4623","Type":"ContainerStarted","Data":"1e49a6e6817c80809222b0fe9f8557e2e86f1ef1c4a1dffce4ca5db33f5b9fc1"}
Jul 15 10:47:11.132731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.132671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:11.373251 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.373194    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:11.560726 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.560678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:11.658873 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:11.658789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:12.370295 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:12.370266    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:12.466749 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:12.466679    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:13.115707 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.115675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr" event={"ID":"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd","Type":"ContainerStarted","Data":"70f515db34a8e893380ac8c1888af5519bcc9a66c889ca1360a018ac37d986d7"}
Jul 15 10:47:13.116083 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.115737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr" event={"ID":"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd","Type":"ContainerStarted","Data":"b3f784e7000060b872a33326a2b96860303d899c33a04de5df922c046404fc32"}
Jul 15 10:47:13.116083 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.115763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr" event={"ID":"7f3a4ff0-1cd9-489a-9c26-1b68750d76bd","Type":"ContainerStarted","Data":"080aef39e2167a59c1379e84e8efcaddcb6d6b6a2d2114326ba8430aae449d77"}
Jul 15 10:47:13.117263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.117238    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9cab10-b113-42f4-9bf7-461420db4623" containerID="610a3ded81e46be9dc9ee849160c25cdff7073c32bfc791ced6d14cf6c5e0b63" exitCode=0
Jul 15 10:47:13.117357 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.117317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nr7q" event={"ID":"ce9cab10-b113-42f4-9bf7-461420db4623","Type":"ContainerDied","Data":"610a3ded81e46be9dc9ee849160c25cdff7073c32bfc791ced6d14cf6c5e0b63"}
Jul 15 10:47:13.235860 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.235818    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-dpnhr" podStartSLOduration=1.768668863 podStartE2EDuration="3.235802291s" totalImagesPullingTime="1.467133428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:13.180870823 +0000 UTC m=+99.015615292" watchObservedRunningTime="2026-07-15 10:47:13.235802291 +0000 UTC m=+99.070546741"
Jul 15 10:47:13.738603 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.738568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"]
Jul 15 10:47:13.741297 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.741276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.745276 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.745251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 15 10:47:13.746216 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 15 10:47:13.746216 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m5kvk\""
Jul 15 10:47:13.746216 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 15 10:47:13.746216 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746137    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4rl5t97plqtu\""
Jul 15 10:47:13.746513 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746499    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 15 10:47:13.746729 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.746716    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 15 10:47:13.764266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.764244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"]
Jul 15 10:47:13.827389 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.827347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:13.917352 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917520 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917520 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-grpc-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917520 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06bd7d9-11bd-4acb-b690-913112365ae7-metrics-client-ca\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917520 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917520 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917734 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgff\" (UniqueName: \"kubernetes.io/projected/a06bd7d9-11bd-4acb-b690-913112365ae7-kube-api-access-2tgff\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:13.917734 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:13.917665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018113 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018113 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-grpc-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018113 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06bd7d9-11bd-4acb-b690-913112365ae7-metrics-client-ca\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018113 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018451 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018451 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgff\" (UniqueName: \"kubernetes.io/projected/a06bd7d9-11bd-4acb-b690-913112365ae7-kube-api-access-2tgff\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018451 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.018451 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.018334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.019505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.019477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06bd7d9-11bd-4acb-b690-913112365ae7-metrics-client-ca\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021061 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.020949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021194 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.021152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021372 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.021345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021601 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.021579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.021616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-grpc-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.021705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.021684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a06bd7d9-11bd-4acb-b690-913112365ae7-secret-thanos-querier-tls\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.037824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.037794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgff\" (UniqueName: \"kubernetes.io/projected/a06bd7d9-11bd-4acb-b690-913112365ae7-kube-api-access-2tgff\") pod \"thanos-querier-5fcf7bdd76-5gcrx\" (UID: \"a06bd7d9-11bd-4acb-b690-913112365ae7\") " pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.051513 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.051480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:14.056241 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.056198    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:14.125099 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.125020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nr7q" event={"ID":"ce9cab10-b113-42f4-9bf7-461420db4623","Type":"ContainerStarted","Data":"304d6c94351b5eaa903941bb1c126bea9718ebc9ffaf682f79a0750ceb23f5c2"}
Jul 15 10:47:14.125407 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.125105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nr7q" event={"ID":"ce9cab10-b113-42f4-9bf7-461420db4623","Type":"ContainerStarted","Data":"0c3aaea26919f80653d9a084e80c63c614ccf04a2469ad5a2eee331b46124aad"}
Jul 15 10:47:14.259117 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.259067    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2nr7q" podStartSLOduration=2.119936406 podStartE2EDuration="4.25902558s" totalImagesPullingTime="2.065471894s" totalInitContainerRuntime="73.61728ms" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:14.207060734 +0000 UTC m=+100.041805194" watchObservedRunningTime="2026-07-15 10:47:14.25902558 +0000 UTC m=+100.093770101"
Jul 15 10:47:14.259608 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.259587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"]
Jul 15 10:47:14.262928 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:14.262908    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06bd7d9_11bd_4acb_b690_913112365ae7.slice/crio-c29f89aed0c5c04922d463d4fc4463a7541297c38683b6f3aab01db3940dac49 WatchSource:0}: Error finding container c29f89aed0c5c04922d463d4fc4463a7541297c38683b6f3aab01db3940dac49: Status 404 returned error can't find the container with id c29f89aed0c5c04922d463d4fc4463a7541297c38683b6f3aab01db3940dac49
Jul 15 10:47:14.414258 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.414228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:14.975803 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.975768    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85777c75db-qjnrg"]
Jul 15 10:47:14.978182 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.978160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:14.980976 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.980939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 15 10:47:14.983809 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.983786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-41nch1lr6l7oo\""
Jul 15 10:47:14.984232 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.984214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 15 10:47:14.984232 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.984226    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bk9p2\""
Jul 15 10:47:14.984400 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.984272    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 15 10:47:14.984460 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:14.984422    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 15 10:47:15.010645 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.010582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85777c75db-qjnrg"]
Jul 15 10:47:15.127761 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.127728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8a61223d-ad4a-4dde-8886-863969e53b20-audit-log\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.127778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzkj\" (UniqueName: \"kubernetes.io/projected/8a61223d-ad4a-4dde-8886-863969e53b20-kube-api-access-dnzkj\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.127855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-client-certs\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.127930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-tls\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.127979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-metrics-server-audit-profiles\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.128020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.128218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.128073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-client-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.134795 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.134762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"c29f89aed0c5c04922d463d4fc4463a7541297c38683b6f3aab01db3940dac49"}
Jul 15 10:47:15.228589 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8a61223d-ad4a-4dde-8886-863969e53b20-audit-log\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228589 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzkj\" (UniqueName: \"kubernetes.io/projected/8a61223d-ad4a-4dde-8886-863969e53b20-kube-api-access-dnzkj\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-client-certs\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-tls\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-metrics-server-audit-profiles\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228945 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-client-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.228974 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.228940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8a61223d-ad4a-4dde-8886-863969e53b20-audit-log\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.230242 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.229987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.230242 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.230197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8a61223d-ad4a-4dde-8886-863969e53b20-metrics-server-audit-profiles\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.232079 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.232035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-client-ca-bundle\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.232165 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.232033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-tls\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.232165 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.232132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8a61223d-ad4a-4dde-8886-863969e53b20-secret-metrics-server-client-certs\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.248426 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.248403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzkj\" (UniqueName: \"kubernetes.io/projected/8a61223d-ad4a-4dde-8886-863969e53b20-kube-api-access-dnzkj\") pod \"metrics-server-85777c75db-qjnrg\" (UID: \"8a61223d-ad4a-4dde-8886-863969e53b20\") " pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.262491 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.262462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:15.290269 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.290228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:15.303847 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.303817    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"]
Jul 15 10:47:15.307140 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.307116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.311231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.311212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 15 10:47:15.311570 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.311552    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-56zlj\""
Jul 15 10:47:15.323764 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.323741    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:47:15.325959 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.325937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.330134 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330114    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 15 10:47:15.330244 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 15 10:47:15.330244 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330114    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 15 10:47:15.330680 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330643    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 15 10:47:15.330783 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f77v6\""
Jul 15 10:47:15.330783 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 15 10:47:15.331020 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 15 10:47:15.331020 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.330914    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 15 10:47:15.335637 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.335617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"]
Jul 15 10:47:15.337487 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.337469    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 15 10:47:15.363028 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.361190    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:47:15.423720 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.423687    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:15.431004 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.430972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431214 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdsv\" (UniqueName: \"kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431214 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431353 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431485 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431545 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhp52\" (UniqueName: \"kubernetes.io/projected/37d80fc6-8e87-4891-ba49-e6c0d345be2d-kube-api-access-zhp52\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.431545 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.431622 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.431560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37d80fc6-8e87-4891-ba49-e6c0d345be2d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.475206 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.475128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85777c75db-qjnrg"]
Jul 15 10:47:15.478245 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:15.478198    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a61223d_ad4a_4dde_8886_863969e53b20.slice/crio-6c5cb22fe763fb0841a5699c4bdab0d4f7f73add414eb1c74f789ba31f07cdec WatchSource:0}: Error finding container 6c5cb22fe763fb0841a5699c4bdab0d4f7f73add414eb1c74f789ba31f07cdec: Status 404 returned error can't find the container with id 6c5cb22fe763fb0841a5699c4bdab0d4f7f73add414eb1c74f789ba31f07cdec
Jul 15 10:47:15.532823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.532823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.532823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhp52\" (UniqueName: \"kubernetes.io/projected/37d80fc6-8e87-4891-ba49-e6c0d345be2d-kube-api-access-zhp52\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.532823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.533228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37d80fc6-8e87-4891-ba49-e6c0d345be2d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.533228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.533228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdsv\" (UniqueName: \"kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.533228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.533228 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.532985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.534514 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.534486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.534638 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.534503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.536503 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.535225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.537161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.537111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.537747 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.537669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.538372 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.538331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.538523 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.538493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37d80fc6-8e87-4891-ba49-e6c0d345be2d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.550629 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.550593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdsv\" (UniqueName: \"kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv\") pod \"console-689fd46d56-92npr\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") " pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:15.551552 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.551524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhp52\" (UniqueName: \"kubernetes.io/projected/37d80fc6-8e87-4891-ba49-e6c0d345be2d-kube-api-access-zhp52\") pod \"monitoring-plugin-5f68df7f58-fmh5p\" (UID: \"37d80fc6-8e87-4891-ba49-e6c0d345be2d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.620535 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.620498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:15.627474 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.627448    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:15.636686 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:15.636657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:16.140141 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:16.140102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg" event={"ID":"8a61223d-ad4a-4dde-8886-863969e53b20","Type":"ContainerStarted","Data":"6c5cb22fe763fb0841a5699c4bdab0d4f7f73add414eb1c74f789ba31f07cdec"}
Jul 15 10:47:16.225457 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:16.225413    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:47:16.231012 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:16.230982    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ffa6a8_9b3c_4d40_8278_ddabfe2220ce.slice/crio-160c12930323af76d31bdafd7c0f9b8136b99e87169e5c9e9136d7ed4758a04a WatchSource:0}: Error finding container 160c12930323af76d31bdafd7c0f9b8136b99e87169e5c9e9136d7ed4758a04a: Status 404 returned error can't find the container with id 160c12930323af76d31bdafd7c0f9b8136b99e87169e5c9e9136d7ed4758a04a
Jul 15 10:47:16.253932 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:16.253902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"]
Jul 15 10:47:16.256931 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:16.256904    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d80fc6_8e87_4891_ba49_e6c0d345be2d.slice/crio-b0840cd154803f639142dcdc9f63a0dc631855410e758fe958951096a6d1091f WatchSource:0}: Error finding container b0840cd154803f639142dcdc9f63a0dc631855410e758fe958951096a6d1091f: Status 404 returned error can't find the container with id b0840cd154803f639142dcdc9f63a0dc631855410e758fe958951096a6d1091f
Jul 15 10:47:17.155077 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.154861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"648b9795cb5dbcf29dcb9d816eefeaefaf7839f65f5e6f77770c95b973428ce9"}
Jul 15 10:47:17.155077 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.154957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"4fb3919b9c615ddcdba99168f70391e4e1f17e30a118d4fd389d94e44c632e1f"}
Jul 15 10:47:17.155077 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.154998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"8e11f6b88f6f8e7229833cd78e29692a348658dfd96114d9e80f617385058dd2"}
Jul 15 10:47:17.157017 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.156984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689fd46d56-92npr" event={"ID":"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce","Type":"ContainerStarted","Data":"160c12930323af76d31bdafd7c0f9b8136b99e87169e5c9e9136d7ed4758a04a"}
Jul 15 10:47:17.164166 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.164120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p" event={"ID":"37d80fc6-8e87-4891-ba49-e6c0d345be2d","Type":"ContainerStarted","Data":"b0840cd154803f639142dcdc9f63a0dc631855410e758fe958951096a6d1091f"}
Jul 15 10:47:17.167102 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.167078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg" event={"ID":"8a61223d-ad4a-4dde-8886-863969e53b20","Type":"ContainerStarted","Data":"57ae380148766a29fb08382feed3f51e3e6e0de3082178366dcc7cf690ed026b"}
Jul 15 10:47:17.201135 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.201038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:17.384664 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.384529    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:17.485578 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.485458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:17.586472 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.586434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:17.784246 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.784156    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:17.997508 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:17.997478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mp8nx"
Jul 15 10:47:18.021004 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:18.020953    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg" podStartSLOduration=2.464150247 podStartE2EDuration="4.020933173s" totalImagesPullingTime="1.556782926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:17.214181776 +0000 UTC m=+103.048926253" watchObservedRunningTime="2026-07-15 10:47:18.020933173 +0000 UTC m=+103.855677623"
Jul 15 10:47:19.029301 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:19.029263    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:19.096760 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:19.096726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:19.183176 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:19.183125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"a835b682daac7a9f9137c93caabd30e90b6e099f32de204b9de33d765365afc3"}
Jul 15 10:47:19.856998 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:19.856958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:20.006455 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.006425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:20.097776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.097749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:20.189563 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.189536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"305e5bcd77c6e623a1f262729461cfb9b910da84cf3ccd4e9dfee07f99ae5e71"}
Jul 15 10:47:20.190909 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.190878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p" event={"ID":"37d80fc6-8e87-4891-ba49-e6c0d345be2d","Type":"ContainerStarted","Data":"f0abdc09774ab2f472f240ed40f98125ea6ef2ae886b61de945251c3b77b99ad"}
Jul 15 10:47:20.191098 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.191073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:20.195826 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.195805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p"
Jul 15 10:47:20.211756 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:20.211706    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-fmh5p" podStartSLOduration=1.429118568 podStartE2EDuration="5.211691884s" totalImagesPullingTime="3.782573316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:20.210815236 +0000 UTC m=+106.045559706" watchObservedRunningTime="2026-07-15 10:47:20.211691884 +0000 UTC m=+106.046436378"
Jul 15 10:47:21.197546 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.197510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" event={"ID":"a06bd7d9-11bd-4acb-b690-913112365ae7","Type":"ContainerStarted","Data":"6b5d3adacb5366ab901b660b14074fbd1398584ae0f0569ce0a99e2d1aaf92a2"}
Jul 15 10:47:21.198018 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.197704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:21.199266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.199242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689fd46d56-92npr" event={"ID":"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce","Type":"ContainerStarted","Data":"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"}
Jul 15 10:47:21.203315 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.203295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx"
Jul 15 10:47:21.231382 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.231325    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fcf7bdd76-5gcrx" podStartSLOduration=4.229506151 podStartE2EDuration="8.231306519s" totalImagesPullingTime="4.001800368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:21.230249234 +0000 UTC m=+107.064993685" watchObservedRunningTime="2026-07-15 10:47:21.231306519 +0000 UTC m=+107.066051021"
Jul 15 10:47:21.280739 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:21.280676    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689fd46d56-92npr" podStartSLOduration=2.135036128 podStartE2EDuration="6.280658286s" totalImagesPullingTime="4.145622158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:21.278864115 +0000 UTC m=+107.113608565" watchObservedRunningTime="2026-07-15 10:47:21.280658286 +0000 UTC m=+107.115402736"
Jul 15 10:47:22.228868 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:22.228832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:22.295910 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:22.295881    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:25.637584 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.637545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:25.637949 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.637639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:25.641928 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.641909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:25.659169 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.659140    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:25.778733 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.778676    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:25.879210 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:25.879177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:26.221197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:26.221168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:47:26.395584 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:26.395546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:26.498888 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:26.498810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:26.601274 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:26.601235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:27.221702 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:27.221671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:32.539712 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:32.539675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:32.638504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:32.638472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:35.291058 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:35.290999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:35.291433 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:35.291094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:35.323082 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:35.323019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:36.275838 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:36.275804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:36.350885 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:36.350848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:42.924736 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:42.924701    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:42.986202 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:42.986174    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:44.482358 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:44.482317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:47:44.484815 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:44.484790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3b89-0a7c-48b6-a769-d93e1d61fe48-metrics-certs\") pod \"network-metrics-daemon-kkfx8\" (UID: \"810a3b89-0a7c-48b6-a769-d93e1d61fe48\") " pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:47:44.584392 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:44.584360    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfsm8\""
Jul 15 10:47:44.592636 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:44.592616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkfx8"
Jul 15 10:47:44.776969 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:44.776867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkfx8"]
Jul 15 10:47:44.779661 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:47:44.779636    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810a3b89_0a7c_48b6_a769_d93e1d61fe48.slice/crio-b07bf8b07ed03ef3f1cc2def658d7bd4787b85b6e8d55249dd57a0cd11763696 WatchSource:0}: Error finding container b07bf8b07ed03ef3f1cc2def658d7bd4787b85b6e8d55249dd57a0cd11763696: Status 404 returned error can't find the container with id b07bf8b07ed03ef3f1cc2def658d7bd4787b85b6e8d55249dd57a0cd11763696
Jul 15 10:47:45.287492 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:45.287451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkfx8" event={"ID":"810a3b89-0a7c-48b6-a769-d93e1d61fe48","Type":"ContainerStarted","Data":"b07bf8b07ed03ef3f1cc2def658d7bd4787b85b6e8d55249dd57a0cd11763696"}
Jul 15 10:47:47.297768 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.297732    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc11f264-bef2-4fd9-92ba-16f892918e19" containerID="ac21299849b5112acbae3a66ddba58285f1d054f38354eb9242c33e0f6f65043" exitCode=0
Jul 15 10:47:47.298190 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.297810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct" event={"ID":"bc11f264-bef2-4fd9-92ba-16f892918e19","Type":"ContainerDied","Data":"ac21299849b5112acbae3a66ddba58285f1d054f38354eb9242c33e0f6f65043"}
Jul 15 10:47:47.298247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.298219    2565 scope.go:117] "RemoveContainer" containerID="ac21299849b5112acbae3a66ddba58285f1d054f38354eb9242c33e0f6f65043"
Jul 15 10:47:47.299731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.299696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkfx8" event={"ID":"810a3b89-0a7c-48b6-a769-d93e1d61fe48","Type":"ContainerStarted","Data":"4486e4643d536f1284caaafa5dc704e597ac82c87366fd3d847ff0c467f38774"}
Jul 15 10:47:47.299803 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.299748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkfx8" event={"ID":"810a3b89-0a7c-48b6-a769-d93e1d61fe48","Type":"ContainerStarted","Data":"e09502afb9f9e60ce17f042be733bb49fe16ebbc22455011d00192907def169a"}
Jul 15 10:47:47.331090 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.331024    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kkfx8" podStartSLOduration=130.691811943 podStartE2EDuration="2m12.331008923s" totalImagesPullingTime="1.63919698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:47.329351617 +0000 UTC m=+133.164096065" watchObservedRunningTime="2026-07-15 10:47:47.331008923 +0000 UTC m=+133.165753372"
Jul 15 10:47:47.424375 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.424339    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:47.526226 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.526191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:47.585682 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:47.585654    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:47:48.306240 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:48.306205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wwxct" event={"ID":"bc11f264-bef2-4fd9-92ba-16f892918e19","Type":"ContainerStarted","Data":"b499cace653d22410014de635a912ecc63887e5f179864b76e9a7eacd74fb0b3"}
Jul 15 10:47:55.296415 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:55.296386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:47:55.300262 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:47:55.300245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85777c75db-qjnrg"
Jul 15 10:48:15.695323 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:15.695291    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:30.197008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:30.196929    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:30.538695 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:30.538554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:39.421504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.421471    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:48:39.425676 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.424318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.438529 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.438501    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:48:39.488458 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.488422    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:39.543690 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.543690 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.544010 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwqm\" (UniqueName: \"kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.544010 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.544010 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.544010 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.544010 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.543883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.577378 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.577350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:39.644369 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwqm\" (UniqueName: \"kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.644926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.644606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.645274 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.645248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.645612 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.645590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.645711 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.645657    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.645711 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.645658    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:39.645797 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.645710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.647898 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.647879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.647987 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.647885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.657916 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.657888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwqm\" (UniqueName: \"kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm\") pod \"console-699476b44b-sk48q\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") " pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.739337 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.739240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:39.891668 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:39.890933    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:48:40.223507 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:40.223466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:40.511593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:40.511491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699476b44b-sk48q" event={"ID":"df9f56db-050e-412e-8611-fbffa1545c7f","Type":"ContainerStarted","Data":"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"}
Jul 15 10:48:40.511593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:40.511540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699476b44b-sk48q" event={"ID":"df9f56db-050e-412e-8611-fbffa1545c7f","Type":"ContainerStarted","Data":"18d0b8baf7a42934368adb01a7cc9013ecae812787a94b1c222a2fe646dbfccf"}
Jul 15 10:48:40.530057 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:40.529985    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-699476b44b-sk48q" podStartSLOduration=1.529972598 podStartE2EDuration="1.529972598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:48:40.528932113 +0000 UTC m=+186.363676590" watchObservedRunningTime="2026-07-15 10:48:40.529972598 +0000 UTC m=+186.364717048"
Jul 15 10:48:49.690543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:49.690506    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:49.739558 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:49.739525    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:49.739693 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:49.739627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:49.744398 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:49.744379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:50.553504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:50.553478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:48:50.603151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:50.603115    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:48:50.723598 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:50.723561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:50.822343 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:50.822316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:48:50.881339 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:48:50.881307    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:15.628038 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.627973    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-689fd46d56-92npr" podUID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" containerName="console" containerID="cri-o://45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34" gracePeriod=15
Jul 15 10:49:15.868516 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.868492    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:49:15.944197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkdsv\" (UniqueName: \"kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944484 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944484 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944484 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944484 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert\") pod \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\" (UID: \"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce\") "
Jul 15 10:49:15.944708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944661    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:15.944708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:15.944802 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config" (OuterVolumeSpecName: "console-config") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:15.944802 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.944720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:49:15.946680 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.946650    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:49:15.946794 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.946684    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:49:15.946794 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:15.946717    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv" (OuterVolumeSpecName: "kube-api-access-jkdsv") pod "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" (UID: "01ffa6a8-9b3c-4d40-8278-ddabfe2220ce"). InnerVolumeSpecName "kube-api-access-jkdsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:49:16.045386 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045345    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045386 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045375    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdsv\" (UniqueName: \"kubernetes.io/projected/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-kube-api-access-jkdsv\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045386 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045385    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045386 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045395    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045404    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045413    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.045651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.045420    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:49:16.656753 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.656715    2565 generic.go:356] "Generic (PLEG): container finished" podID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" containerID="45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34" exitCode=0
Jul 15 10:49:16.657191 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.656761    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689fd46d56-92npr"
Jul 15 10:49:16.657191 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.656807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689fd46d56-92npr" event={"ID":"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce","Type":"ContainerDied","Data":"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"}
Jul 15 10:49:16.657191 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.656866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689fd46d56-92npr" event={"ID":"01ffa6a8-9b3c-4d40-8278-ddabfe2220ce","Type":"ContainerDied","Data":"160c12930323af76d31bdafd7c0f9b8136b99e87169e5c9e9136d7ed4758a04a"}
Jul 15 10:49:16.657191 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.656890    2565 scope.go:117] "RemoveContainer" containerID="45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"
Jul 15 10:49:16.665560 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.665537    2565 scope.go:117] "RemoveContainer" containerID="45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"
Jul 15 10:49:16.665858 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:49:16.665836    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34\": container with ID starting with 45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34 not found: ID does not exist" containerID="45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"
Jul 15 10:49:16.665911 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.665867    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34"} err="failed to get container status \"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34\": rpc error: code = NotFound desc = could not find container \"45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34\": container with ID starting with 45d24e1022667f7daa8521581884ff3dd98f41a039d474fc711390074341db34 not found: ID does not exist"
Jul 15 10:49:16.679199 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.679174    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:49:16.682275 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:16.682252    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-689fd46d56-92npr"]
Jul 15 10:49:18.672521 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:18.672485    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" path="/var/lib/kubelet/pods/01ffa6a8-9b3c-4d40-8278-ddabfe2220ce/volumes"
Jul 15 10:49:35.172890 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:35.172852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:35.254778 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:35.254725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:45.483254 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:45.483219    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:45.542632 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:45.542594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:48.169783 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.169750    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:49:48.170156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.170061    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" containerName="console"
Jul 15 10:49:48.170156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.170082    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" containerName="console"
Jul 15 10:49:48.170156 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.170139    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ffa6a8-9b3c-4d40-8278-ddabfe2220ce" containerName="console"
Jul 15 10:49:48.174383 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.174354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.186607 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.186583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:49:48.236296 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.236258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:48.302456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302603 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302603 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302603 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kn4v\" (UniqueName: \"kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302603 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302732 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.302732 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.302692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.306183 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.306161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:48.404127 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kn4v\" (UniqueName: \"kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.404291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.405060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.405060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.404985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.405242 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.405140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.405242 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.405212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.406755 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.406725    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.407281 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.407255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.413086 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.413065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kn4v\" (UniqueName: \"kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v\") pod \"console-7cb99c7fd9-qqhvd\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") " pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.484926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.484844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:48.573725 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.573692    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:48.622296 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.622271    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:49:48.625460 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:49:48.625413    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50649d4b_0874_4cac_b0f0_8760c4a9e090.slice/crio-18180017310c3cb806586c98f26b5a29e81f08d2744d3cbf71f57622f713a4e1 WatchSource:0}: Error finding container 18180017310c3cb806586c98f26b5a29e81f08d2744d3cbf71f57622f713a4e1: Status 404 returned error can't find the container with id 18180017310c3cb806586c98f26b5a29e81f08d2744d3cbf71f57622f713a4e1
Jul 15 10:49:48.674250 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.674227    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:48.778798 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.778702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99c7fd9-qqhvd" event={"ID":"50649d4b-0874-4cac-b0f0-8760c4a9e090","Type":"ContainerStarted","Data":"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"}
Jul 15 10:49:48.778798 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.778749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99c7fd9-qqhvd" event={"ID":"50649d4b-0874-4cac-b0f0-8760c4a9e090","Type":"ContainerStarted","Data":"18180017310c3cb806586c98f26b5a29e81f08d2744d3cbf71f57622f713a4e1"}
Jul 15 10:49:48.797327 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:48.797271    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cb99c7fd9-qqhvd" podStartSLOduration=0.797254934 podStartE2EDuration="797.254934ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:49:48.796933976 +0000 UTC m=+254.631678449" watchObservedRunningTime="2026-07-15 10:49:48.797254934 +0000 UTC m=+254.631999384"
Jul 15 10:49:49.170146 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:49.170113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:49.228283 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:49.228252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:57.450430 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:57.450348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:57.512060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:57.512021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:58.485583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:58.485528    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:58.485583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:58.485598    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:58.490479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:58.490456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:58.821323 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:58.821241    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:49:58.868826 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:58.868789    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:49:59.025703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:59.025661    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:49:59.126264 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:49:59.126227    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:50:02.546406 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:02.546371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:50:02.642965 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:02.642932    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:50:11.615177 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:11.615137    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:50:23.893625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:23.893566    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-699476b44b-sk48q" podUID="df9f56db-050e-412e-8611-fbffa1545c7f" containerName="console" containerID="cri-o://c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611" gracePeriod=15
Jul 15 10:50:24.131293 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.131267    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:50:24.186083 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.185970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzwqm\" (UniqueName: \"kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186083 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186083 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle\") pod \"df9f56db-050e-412e-8611-fbffa1545c7f\" (UID: \"df9f56db-050e-412e-8611-fbffa1545c7f\") "
Jul 15 10:50:24.186549 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:50:24.186663 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186636    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config" (OuterVolumeSpecName: "console-config") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:50:24.186719 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186643    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:50:24.186774 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.186714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:50:24.188439 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.188410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:50:24.188552 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.188434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:50:24.188552 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.188466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm" (OuterVolumeSpecName: "kube-api-access-hzwqm") pod "df9f56db-050e-412e-8611-fbffa1545c7f" (UID: "df9f56db-050e-412e-8611-fbffa1545c7f"). InnerVolumeSpecName "kube-api-access-hzwqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:50:24.287705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287667    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287700    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287711    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287705 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287719    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287940 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287727    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwqm\" (UniqueName: \"kubernetes.io/projected/df9f56db-050e-412e-8611-fbffa1545c7f-kube-api-access-hzwqm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287940 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287736    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df9f56db-050e-412e-8611-fbffa1545c7f-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.287940 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.287743    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df9f56db-050e-412e-8611-fbffa1545c7f-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:50:24.915583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.915550    2565 generic.go:356] "Generic (PLEG): container finished" podID="df9f56db-050e-412e-8611-fbffa1545c7f" containerID="c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611" exitCode=0
Jul 15 10:50:24.915989 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.915598    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699476b44b-sk48q"
Jul 15 10:50:24.915989 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.915618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699476b44b-sk48q" event={"ID":"df9f56db-050e-412e-8611-fbffa1545c7f","Type":"ContainerDied","Data":"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"}
Jul 15 10:50:24.915989 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.915663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699476b44b-sk48q" event={"ID":"df9f56db-050e-412e-8611-fbffa1545c7f","Type":"ContainerDied","Data":"18d0b8baf7a42934368adb01a7cc9013ecae812787a94b1c222a2fe646dbfccf"}
Jul 15 10:50:24.915989 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.915683    2565 scope.go:117] "RemoveContainer" containerID="c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"
Jul 15 10:50:24.924017 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.924000    2565 scope.go:117] "RemoveContainer" containerID="c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"
Jul 15 10:50:24.924329 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:50:24.924311    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611\": container with ID starting with c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611 not found: ID does not exist" containerID="c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"
Jul 15 10:50:24.924394 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.924337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611"} err="failed to get container status \"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611\": rpc error: code = NotFound desc = could not find container \"c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611\": container with ID starting with c810ed4a770170b0e2899a50b197b246ffe08b6746e16ab3de26beca7cc2b611 not found: ID does not exist"
Jul 15 10:50:24.933174 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.933124    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:50:24.941919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:24.941899    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-699476b44b-sk48q"]
Jul 15 10:50:26.673094 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:26.673060    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9f56db-050e-412e-8611-fbffa1545c7f" path="/var/lib/kubelet/pods/df9f56db-050e-412e-8611-fbffa1545c7f/volumes"
Jul 15 10:50:34.534609 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:34.534582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:50:34.538723 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:34.538693    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:50:34.540762 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:34.540741    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:50:34.545187 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:34.545167    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 15 10:50:34.545279 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:50:34.545231    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:51:04.142714 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:04.142682    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:05.455231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.455194    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"]
Jul 15 10:51:05.455596 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.455517    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9f56db-050e-412e-8611-fbffa1545c7f" containerName="console"
Jul 15 10:51:05.455596 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.455527    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9f56db-050e-412e-8611-fbffa1545c7f" containerName="console"
Jul 15 10:51:05.455596 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.455578    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9f56db-050e-412e-8611-fbffa1545c7f" containerName="console"
Jul 15 10:51:05.458326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.458307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.460883 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.460860    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bl6lr\""
Jul 15 10:51:05.461062 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.461021    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 15 10:51:05.461676 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.461650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 15 10:51:05.473178 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.473153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"]
Jul 15 10:51:05.494579 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.494550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:05.614879 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.614838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.615081 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.614899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.615081 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.614952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pdm\" (UniqueName: \"kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.715606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.715540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.715606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.715573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pdm\" (UniqueName: \"kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.715725 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.715626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.715959 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.715940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.716008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.715988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.724731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.724700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pdm\" (UniqueName: \"kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.767519 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.767499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:05.898629 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.898604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"]
Jul 15 10:51:05.901085 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:51:05.901034    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8deab7_b83a_4f63_8649_48026a452084.slice/crio-124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce WatchSource:0}: Error finding container 124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce: Status 404 returned error can't find the container with id 124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce
Jul 15 10:51:05.902939 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:05.902918    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:51:06.072306 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:06.072274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss" event={"ID":"ad8deab7-b83a-4f63-8649-48026a452084","Type":"ContainerStarted","Data":"124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce"}
Jul 15 10:51:12.099736 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:12.099698    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad8deab7-b83a-4f63-8649-48026a452084" containerID="a5ba95cf66427a0a068af2239d20a33ea9d576e5d4aae22e93eda92414e909be" exitCode=0
Jul 15 10:51:12.100305 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:12.099787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss" event={"ID":"ad8deab7-b83a-4f63-8649-48026a452084","Type":"ContainerDied","Data":"a5ba95cf66427a0a068af2239d20a33ea9d576e5d4aae22e93eda92414e909be"}
Jul 15 10:51:15.116664 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:15.116616    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad8deab7-b83a-4f63-8649-48026a452084" containerID="ee6204c2976956bb922dc365ee7067e14fb1ad969d1dbc32777f4879524fc814" exitCode=0
Jul 15 10:51:15.117029 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:15.116678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss" event={"ID":"ad8deab7-b83a-4f63-8649-48026a452084","Type":"ContainerDied","Data":"ee6204c2976956bb922dc365ee7067e14fb1ad969d1dbc32777f4879524fc814"}
Jul 15 10:51:22.155841 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:22.155807    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad8deab7-b83a-4f63-8649-48026a452084" containerID="94579f0fea1ad8706b85703dbd0719920d35f66a39d482060a4f879502be1ded" exitCode=0
Jul 15 10:51:22.156396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:22.155874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss" event={"ID":"ad8deab7-b83a-4f63-8649-48026a452084","Type":"ContainerDied","Data":"94579f0fea1ad8706b85703dbd0719920d35f66a39d482060a4f879502be1ded"}
Jul 15 10:51:23.281549 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.281525    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:23.370531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.370436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle\") pod \"ad8deab7-b83a-4f63-8649-48026a452084\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") "
Jul 15 10:51:23.370531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.370507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util\") pod \"ad8deab7-b83a-4f63-8649-48026a452084\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") "
Jul 15 10:51:23.370531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.370530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46pdm\" (UniqueName: \"kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm\") pod \"ad8deab7-b83a-4f63-8649-48026a452084\" (UID: \"ad8deab7-b83a-4f63-8649-48026a452084\") "
Jul 15 10:51:23.371151 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.371120    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle" (OuterVolumeSpecName: "bundle") pod "ad8deab7-b83a-4f63-8649-48026a452084" (UID: "ad8deab7-b83a-4f63-8649-48026a452084"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:51:23.372813 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.372792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm" (OuterVolumeSpecName: "kube-api-access-46pdm") pod "ad8deab7-b83a-4f63-8649-48026a452084" (UID: "ad8deab7-b83a-4f63-8649-48026a452084"). InnerVolumeSpecName "kube-api-access-46pdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:51:23.374679 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.374654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util" (OuterVolumeSpecName: "util") pod "ad8deab7-b83a-4f63-8649-48026a452084" (UID: "ad8deab7-b83a-4f63-8649-48026a452084"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:51:23.471309 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.471240    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:51:23.471309 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.471266    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pdm\" (UniqueName: \"kubernetes.io/projected/ad8deab7-b83a-4f63-8649-48026a452084-kube-api-access-46pdm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:51:23.471309 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:23.471297    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad8deab7-b83a-4f63-8649-48026a452084-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:51:24.165415 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:24.165379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss" event={"ID":"ad8deab7-b83a-4f63-8649-48026a452084","Type":"ContainerDied","Data":"124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce"}
Jul 15 10:51:24.165576 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:24.165435    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qppss"
Jul 15 10:51:24.165576 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:24.165437    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124fa39b04ff8b6b9d490576bda94962d74f181e5d0b02a7317bf79eae2c48ce"
Jul 15 10:51:26.689011 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:26.688973    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:27.051466 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:27.051381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:28.470277 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470242    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"]
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470554    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="util"
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470566    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="util"
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470576    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="pull"
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470581    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="pull"
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470595    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="extract"
Jul 15 10:51:28.470631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470601    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="extract"
Jul 15 10:51:28.470823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.470657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8deab7-b83a-4f63-8649-48026a452084" containerName="extract"
Jul 15 10:51:28.515736 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.515702    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"]
Jul 15 10:51:28.515886 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.515806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.521132 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.521097    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ndjh7\""
Jul 15 10:51:28.521280 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.521196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 15 10:51:28.521525 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.521505    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 15 10:51:28.521793 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.521778    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 15 10:51:28.612909 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.612876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wks\" (UniqueName: \"kubernetes.io/projected/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-kube-api-access-m4wks\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.613074 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.612921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.714266 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.714234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wks\" (UniqueName: \"kubernetes.io/projected/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-kube-api-access-m4wks\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.714402 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.714279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.716787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.716755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.731866 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.731795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wks\" (UniqueName: \"kubernetes.io/projected/c2ce75e3-66e5-4229-a11d-c707fa2c0b34-kube-api-access-m4wks\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-stn6l\" (UID: \"c2ce75e3-66e5-4229-a11d-c707fa2c0b34\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.825906 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.825875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:28.984745 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:28.984651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"]
Jul 15 10:51:28.988287 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:51:28.988259    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ce75e3_66e5_4229_a11d_c707fa2c0b34.slice/crio-864b2279a8261f09ec9a9c8d99931486e776138d048d6b6e03ac132f282998a4 WatchSource:0}: Error finding container 864b2279a8261f09ec9a9c8d99931486e776138d048d6b6e03ac132f282998a4: Status 404 returned error can't find the container with id 864b2279a8261f09ec9a9c8d99931486e776138d048d6b6e03ac132f282998a4
Jul 15 10:51:29.187190 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:29.187148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l" event={"ID":"c2ce75e3-66e5-4229-a11d-c707fa2c0b34","Type":"ContainerStarted","Data":"864b2279a8261f09ec9a9c8d99931486e776138d048d6b6e03ac132f282998a4"}
Jul 15 10:51:44.251429 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.251389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l" event={"ID":"c2ce75e3-66e5-4229-a11d-c707fa2c0b34","Type":"ContainerStarted","Data":"146cfcb86ed403f8547832ecb508dd703423c0a5b076c1ca0565fe38e4a6728e"}
Jul 15 10:51:44.251875 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.251592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:51:44.274137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.274089    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l" podStartSLOduration=1.481286074 podStartE2EDuration="16.274068463s" totalImagesPullingTime="14.792782389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:44.271585382 +0000 UTC m=+370.106329843" watchObservedRunningTime="2026-07-15 10:51:44.274068463 +0000 UTC m=+370.108812925"
Jul 15 10:51:44.291334 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.291307    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:44.578884 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.578851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:44.589149 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.589118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-ttcxg"]
Jul 15 10:51:44.591506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.591484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.593789 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.593773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 15 10:51:44.593910 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.593844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 15 10:51:44.593910 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.593869    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-t4wql\""
Jul 15 10:51:44.601776 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.601754    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-ttcxg"]
Jul 15 10:51:44.675435 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.675403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:44.744923 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.744891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.745112 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.744941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcld\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-kube-api-access-qxcld\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.745112 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.745036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-cabundle0\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.807835 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.807806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:44.846159 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.846067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.846159 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.846125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcld\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-kube-api-access-qxcld\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.846388 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.846186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-cabundle0\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.846388 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:44.846218    2565 secret.go:281] references non-existent secret key: ca.crt
Jul 15 10:51:44.846388 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:44.846235    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 15 10:51:44.846388 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:44.846243    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-ttcxg: references non-existent secret key: ca.crt
Jul 15 10:51:44.846388 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:44.846296    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates podName:e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7 nodeName:}" failed. No retries permitted until 2026-07-15 10:51:45.346282081 +0000 UTC m=+371.181026512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates") pod "keda-operator-77f648d694-ttcxg" (UID: "e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7") : references non-existent secret key: ca.crt
Jul 15 10:51:44.846740 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.846723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-cabundle0\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.857427 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.857396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcld\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-kube-api-access-qxcld\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:44.993223 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:44.993189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:45.007593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.007568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"]
Jul 15 10:51:45.009916 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.009897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.012814 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.012789    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 15 10:51:45.022566 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.022541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"]
Jul 15 10:51:45.149568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.149488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-certificates\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.149704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.149585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/89fe2123-95c3-4547-8a68-f82f9b893baa-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.149704 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.149605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4zf\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-kube-api-access-5j4zf\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.242345 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.242310    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-mxj2z"]
Jul 15 10:51:45.242499 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.242308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:45.244569 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.244555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.246898 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.246879    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 15 10:51:45.250391 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.250371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/89fe2123-95c3-4547-8a68-f82f9b893baa-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.250506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.250398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4zf\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-kube-api-access-5j4zf\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.250506 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.250440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-certificates\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.250744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.250727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/89fe2123-95c3-4547-8a68-f82f9b893baa-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.254743 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.254073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-certificates\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.255316 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.255295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-mxj2z"]
Jul 15 10:51:45.260947 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.260920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4zf\" (UniqueName: \"kubernetes.io/projected/89fe2123-95c3-4547-8a68-f82f9b893baa-kube-api-access-5j4zf\") pod \"keda-metrics-apiserver-55557df8c-27v2g\" (UID: \"89fe2123-95c3-4547-8a68-f82f9b893baa\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.307362 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.307334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:45.320666 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.320626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:45.351408 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.351380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.351652 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.351622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcmf\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-kube-api-access-nzcmf\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.351723 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.351680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:45.354440 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.354413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7-certificates\") pod \"keda-operator-77f648d694-ttcxg\" (UID: \"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7\") " pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:45.409179 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.409095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:45.454655 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.453867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcmf\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-kube-api-access-nzcmf\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.454655 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.453945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.454655 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:45.454150    2565 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 15 10:51:45.454655 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:45.454178    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-mxj2z: secret "keda-admission-webhooks-certs" not found
Jul 15 10:51:45.454655 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:51:45.454254    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates podName:e3700a17-8ce9-43b2-ab8a-5b6e50188e55 nodeName:}" failed. No retries permitted until 2026-07-15 10:51:45.954230935 +0000 UTC m=+371.788975387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates") pod "keda-admission-56f9798bb4-mxj2z" (UID: "e3700a17-8ce9-43b2-ab8a-5b6e50188e55") : secret "keda-admission-webhooks-certs" not found
Jul 15 10:51:45.465492 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.465467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcmf\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-kube-api-access-nzcmf\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.481496 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.481472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"]
Jul 15 10:51:45.483699 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:51:45.483663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fe2123_95c3_4547_8a68_f82f9b893baa.slice/crio-3fed4e9392646131488819cee1e1cb88f7d0b7d0c2afd383dc48d8e0dce3c494 WatchSource:0}: Error finding container 3fed4e9392646131488819cee1e1cb88f7d0b7d0c2afd383dc48d8e0dce3c494: Status 404 returned error can't find the container with id 3fed4e9392646131488819cee1e1cb88f7d0b7d0c2afd383dc48d8e0dce3c494
Jul 15 10:51:45.502834 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.502804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:45.660140 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.660020    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-ttcxg"]
Jul 15 10:51:45.662669 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:51:45.662644    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16fbfb9_24d5_4567_8f18_8a1cf5f75ea7.slice/crio-b5994be95c63aee2fd1520b017c7ec77999c6b523f5938dcc7b64fb4340b3060 WatchSource:0}: Error finding container b5994be95c63aee2fd1520b017c7ec77999c6b523f5938dcc7b64fb4340b3060: Status 404 returned error can't find the container with id b5994be95c63aee2fd1520b017c7ec77999c6b523f5938dcc7b64fb4340b3060
Jul 15 10:51:45.662961 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.662935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:45.959380 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.959301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.961967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.961944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3700a17-8ce9-43b2-ab8a-5b6e50188e55-certificates\") pod \"keda-admission-56f9798bb4-mxj2z\" (UID: \"e3700a17-8ce9-43b2-ab8a-5b6e50188e55\") " pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:45.983858 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:45.983830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:46.086706 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:46.086681    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:51:46.156024 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:46.155991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:46.264946 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:46.264910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-ttcxg" event={"ID":"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7","Type":"ContainerStarted","Data":"b5994be95c63aee2fd1520b017c7ec77999c6b523f5938dcc7b64fb4340b3060"}
Jul 15 10:51:46.266844 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:46.266818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g" event={"ID":"89fe2123-95c3-4547-8a68-f82f9b893baa","Type":"ContainerStarted","Data":"3fed4e9392646131488819cee1e1cb88f7d0b7d0c2afd383dc48d8e0dce3c494"}
Jul 15 10:51:46.311062 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:46.311021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-mxj2z"]
Jul 15 10:51:46.313926 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:51:46.313839    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3700a17_8ce9_43b2_ab8a_5b6e50188e55.slice/crio-ba03965f2ce13ff078c3840f6c445e701191c9f7c9b94524761133c1ae4ae4f8 WatchSource:0}: Error finding container ba03965f2ce13ff078c3840f6c445e701191c9f7c9b94524761133c1ae4ae4f8: Status 404 returned error can't find the container with id ba03965f2ce13ff078c3840f6c445e701191c9f7c9b94524761133c1ae4ae4f8
Jul 15 10:51:47.274563 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:47.274528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z" event={"ID":"e3700a17-8ce9-43b2-ab8a-5b6e50188e55","Type":"ContainerStarted","Data":"ba03965f2ce13ff078c3840f6c445e701191c9f7c9b94524761133c1ae4ae4f8"}
Jul 15 10:51:50.291222 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.291178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-ttcxg" event={"ID":"e16fbfb9-24d5-4567-8f18-8a1cf5f75ea7","Type":"ContainerStarted","Data":"d49d841ad6051f649c3cef9bd1ef5c85a189070bdfc104fe34a1f6e553fa9754"}
Jul 15 10:51:50.291663 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.291396    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:51:50.292638 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.292616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g" event={"ID":"89fe2123-95c3-4547-8a68-f82f9b893baa","Type":"ContainerStarted","Data":"e94284bfc02c362a698d424ab0dd7ad54bae2c452ca18e027eba11a27f84b62f"}
Jul 15 10:51:50.292743 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.292669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:51:50.293958 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.293923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z" event={"ID":"e3700a17-8ce9-43b2-ab8a-5b6e50188e55","Type":"ContainerStarted","Data":"91d65e43da0aa4e7490156585ee566e028c4539694b9397f9d4d59b97e804415"}
Jul 15 10:51:50.294035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.293983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:51:50.310455 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.310397    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-ttcxg" podStartSLOduration=2.31520475 podStartE2EDuration="6.310379611s" totalImagesPullingTime="3.995174861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:50.308863586 +0000 UTC m=+376.143608100" watchObservedRunningTime="2026-07-15 10:51:50.310379611 +0000 UTC m=+376.145124062"
Jul 15 10:51:50.327531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.327481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z" podStartSLOduration=1.994081948 podStartE2EDuration="5.327459182s" totalImagesPullingTime="3.333377234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:50.327296486 +0000 UTC m=+376.162040937" watchObservedRunningTime="2026-07-15 10:51:50.327459182 +0000 UTC m=+376.162203633"
Jul 15 10:51:50.345504 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:51:50.345467    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g" podStartSLOduration=2.171727025 podStartE2EDuration="6.345454481s" totalImagesPullingTime="4.173727456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:50.343842357 +0000 UTC m=+376.178586833" watchObservedRunningTime="2026-07-15 10:51:50.345454481 +0000 UTC m=+376.180198930"
Jul 15 10:52:01.302362 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:01.302327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-27v2g"
Jul 15 10:52:05.260970 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:05.260937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-stn6l"
Jul 15 10:52:11.300884 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:11.300851    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-ttcxg"
Jul 15 10:52:11.301341 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:11.300974    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-mxj2z"
Jul 15 10:52:52.118088 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.118015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.227179 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.227150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.230732 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.230704    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 10:52:52.233078 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.233035    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.239094 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.239061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7xrpg\""
Jul 15 10:52:52.239280 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.239263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 15 10:52:52.239696 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.239680    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 15 10:52:52.239762 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.239680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 15 10:52:52.250004 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.249980    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 10:52:52.288334 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.288302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvgj\" (UniqueName: \"kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.288488 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.288359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.342232 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.342200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.389012 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.388942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvgj\" (UniqueName: \"kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.389012 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.388992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.391536 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.391517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.397244 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.397222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvgj\" (UniqueName: \"kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj\") pod \"llmisvc-controller-manager-dd9fc96f-r22s7\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") " pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.444528 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.444479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.548007 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.547974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:52.626275 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.626243    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.682976 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.682887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 10:52:52.685718 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:52:52.685693    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod751a12f4_9aaf_44f8_9c81_b3c223db455e.slice/crio-44b6c292c19a27e43e0a0b777edeafff7fda5e1beac8456f2e1c71d179f8e376 WatchSource:0}: Error finding container 44b6c292c19a27e43e0a0b777edeafff7fda5e1beac8456f2e1c71d179f8e376: Status 404 returned error can't find the container with id 44b6c292c19a27e43e0a0b777edeafff7fda5e1beac8456f2e1c71d179f8e376
Jul 15 10:52:52.848792 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.848758    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:52.948517 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:52.948441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:52:53.568629 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:53.568597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" event={"ID":"751a12f4-9aaf-44f8-9c81-b3c223db455e","Type":"ContainerStarted","Data":"44b6c292c19a27e43e0a0b777edeafff7fda5e1beac8456f2e1c71d179f8e376"}
Jul 15 10:52:55.579627 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:55.579606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:52:55.598507 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:55.598463    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" podStartSLOduration=0.774950629 podStartE2EDuration="3.598448623s" totalImagesPullingTime="2.823497994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:55.597468359 +0000 UTC m=+441.432212833" watchObservedRunningTime="2026-07-15 10:52:55.598448623 +0000 UTC m=+441.433193083"
Jul 15 10:52:56.585486 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:52:56.585441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" event={"ID":"751a12f4-9aaf-44f8-9c81-b3c223db455e","Type":"ContainerStarted","Data":"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"}
Jul 15 10:53:12.108034 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:12.107991    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:27.593287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:27.593254    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 10:53:34.603035 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.603006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:34.707978 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.707935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:34.867890 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.867816    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"]
Jul 15 10:53:34.872496 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.872478    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:34.874909 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.874887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 15 10:53:34.875025 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.874938    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 15 10:53:34.875573 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.875561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bl6lr\""
Jul 15 10:53:34.883555 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.883534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"]
Jul 15 10:53:34.948541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.948507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:34.948541 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.948544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:34.948758 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:34.948579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wddd\" (UniqueName: \"kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.049880 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.049844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.050072 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.049894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wddd\" (UniqueName: \"kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.050072 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.049953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.050405 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.050383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.050447 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.050407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.058231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.058207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wddd\" (UniqueName: \"kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.181913 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.181827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:35.317690 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.317658    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"]
Jul 15 10:53:35.320574 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:53:35.320550    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5259c1e9_ce9b_4fb5_bcc9_d194f08ab40a.slice/crio-f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408 WatchSource:0}: Error finding container f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408: Status 404 returned error can't find the container with id f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408
Jul 15 10:53:35.762030 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.761994    2565 generic.go:356] "Generic (PLEG): container finished" podID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerID="98438ffbb4849d068e65bc134a83032264885def09a9ab2c18beb724e5198c86" exitCode=0
Jul 15 10:53:35.762417 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.762078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft" event={"ID":"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a","Type":"ContainerDied","Data":"98438ffbb4849d068e65bc134a83032264885def09a9ab2c18beb724e5198c86"}
Jul 15 10:53:35.762417 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:35.762122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft" event={"ID":"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a","Type":"ContainerStarted","Data":"f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408"}
Jul 15 10:53:37.774324 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:37.774286    2565 generic.go:356] "Generic (PLEG): container finished" podID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerID="cd34115c17a891194d93b8f9fe5f6480cc08c87034555cf8f9562fe66a416719" exitCode=0
Jul 15 10:53:37.774679 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:37.774377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft" event={"ID":"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a","Type":"ContainerDied","Data":"cd34115c17a891194d93b8f9fe5f6480cc08c87034555cf8f9562fe66a416719"}
Jul 15 10:53:38.781376 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:38.781338    2565 generic.go:356] "Generic (PLEG): container finished" podID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerID="e2c556b32e8f1431f591304498952ec8b721487c18c47265a9cd1169f3630d0a" exitCode=0
Jul 15 10:53:38.781744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:38.781428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft" event={"ID":"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a","Type":"ContainerDied","Data":"e2c556b32e8f1431f591304498952ec8b721487c18c47265a9cd1169f3630d0a"}
Jul 15 10:53:39.921346 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.921322    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:39.996340 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.996304    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wddd\" (UniqueName: \"kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd\") pod \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") "
Jul 15 10:53:39.996531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.996367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle\") pod \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") "
Jul 15 10:53:39.996531 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.996403    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util\") pod \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\" (UID: \"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a\") "
Jul 15 10:53:39.997136 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.997109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle" (OuterVolumeSpecName: "bundle") pod "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" (UID: "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:53:39.998584 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:39.998562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd" (OuterVolumeSpecName: "kube-api-access-2wddd") pod "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" (UID: "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a"). InnerVolumeSpecName "kube-api-access-2wddd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:53:40.002447 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.002420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util" (OuterVolumeSpecName: "util") pod "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" (UID: "5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:53:40.097827 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.097793    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wddd\" (UniqueName: \"kubernetes.io/projected/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-kube-api-access-2wddd\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:53:40.097827 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.097825    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:53:40.097827 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.097835    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:53:40.792505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.792469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft" event={"ID":"5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a","Type":"ContainerDied","Data":"f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408"}
Jul 15 10:53:40.792505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.792498    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m5ft"
Jul 15 10:53:40.792708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:40.792520    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f735b4ac22b543555e72217ff29d2ee1e3de82edefcf4a41fa505a371ad7f408"
Jul 15 10:53:43.855226 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:43.855187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:43.919403 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:43.919364    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:45.796201 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:45.796169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:46.683434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:46.683401    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:47.858023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:47.857990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:49.632665 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:49.632632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:49.732116 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:49.732033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:51.850142 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:51.850109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:51.951262 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:51.951227    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:53.208396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:53.208365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.285911 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.285869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.483668 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.483635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.590101 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.590069    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.673576 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.673544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.700920 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.700882    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:53:54.892979 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.892896    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:54.994308 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:54.994262    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:56.200610 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:56.200575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:56.303864 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:56.303826    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:56.412795 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:56.412761    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:53:57.213815 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.213784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"]
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214148    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="util"
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214159    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="util"
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214180    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="pull"
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214186    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="pull"
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214192    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="extract"
Jul 15 10:53:57.214197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214198    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="extract"
Jul 15 10:53:57.214377 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.214253    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5259c1e9-ce9b-4fb5-bcc9-d194f08ab40a" containerName="extract"
Jul 15 10:53:57.216557 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.216536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.218922 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.218897    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 15 10:53:57.219022 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.218990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 15 10:53:57.219727 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.219712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bl6lr\""
Jul 15 10:53:57.228825 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.228791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"]
Jul 15 10:53:57.343807 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.343768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.343946 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.343834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.343946 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.343921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbg6\" (UniqueName: \"kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.445163 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.445130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.445331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.445196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.445331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.445246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbg6\" (UniqueName: \"kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.445593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.445569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.445631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.445577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.454454 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.454431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbg6\" (UniqueName: \"kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.527286 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.527217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:53:57.664830 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.664799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"]
Jul 15 10:53:57.668336 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:53:57.668296    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc2cde5_218c_48bf_bad6_45ffd19da41b.slice/crio-1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf WatchSource:0}: Error finding container 1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf: Status 404 returned error can't find the container with id 1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf
Jul 15 10:53:57.873928 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.873894    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerID="64b25bf933afaa949a5f469eec206934181e93ea29475c51ee7063f954785491" exitCode=0
Jul 15 10:53:57.874133 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.873976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc" event={"ID":"3fc2cde5-218c-48bf-bad6-45ffd19da41b","Type":"ContainerDied","Data":"64b25bf933afaa949a5f469eec206934181e93ea29475c51ee7063f954785491"}
Jul 15 10:53:57.874133 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:57.874026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc" event={"ID":"3fc2cde5-218c-48bf-bad6-45ffd19da41b","Type":"ContainerStarted","Data":"1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf"}
Jul 15 10:53:59.884895 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:59.884864    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerID="8be0829aeb4ced09d71bad557d055043584edc3f87e3b2c5be32d8da2def7ce7" exitCode=0
Jul 15 10:53:59.885287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:53:59.884930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc" event={"ID":"3fc2cde5-218c-48bf-bad6-45ffd19da41b","Type":"ContainerDied","Data":"8be0829aeb4ced09d71bad557d055043584edc3f87e3b2c5be32d8da2def7ce7"}
Jul 15 10:54:00.891826 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:00.891792    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerID="2e1589896ed2b16851db8e04dd2b7e10b16d4681699880ebaf2f52607897dc84" exitCode=0
Jul 15 10:54:00.892223 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:00.891876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc" event={"ID":"3fc2cde5-218c-48bf-bad6-45ffd19da41b","Type":"ContainerDied","Data":"2e1589896ed2b16851db8e04dd2b7e10b16d4681699880ebaf2f52607897dc84"}
Jul 15 10:54:01.527983 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:01.527946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:02.031503 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.031477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:54:02.189889 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.189802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsbg6\" (UniqueName: \"kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6\") pod \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") "
Jul 15 10:54:02.189889 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.189846    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle\") pod \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") "
Jul 15 10:54:02.190169 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.189931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util\") pod \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\" (UID: \"3fc2cde5-218c-48bf-bad6-45ffd19da41b\") "
Jul 15 10:54:02.190370 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.190343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle" (OuterVolumeSpecName: "bundle") pod "3fc2cde5-218c-48bf-bad6-45ffd19da41b" (UID: "3fc2cde5-218c-48bf-bad6-45ffd19da41b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:02.191995 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.191967    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6" (OuterVolumeSpecName: "kube-api-access-hsbg6") pod "3fc2cde5-218c-48bf-bad6-45ffd19da41b" (UID: "3fc2cde5-218c-48bf-bad6-45ffd19da41b"). InnerVolumeSpecName "kube-api-access-hsbg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:02.195940 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.195914    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util" (OuterVolumeSpecName: "util") pod "3fc2cde5-218c-48bf-bad6-45ffd19da41b" (UID: "3fc2cde5-218c-48bf-bad6-45ffd19da41b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:02.291421 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.291387    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:02.291421 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.291417    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbg6\" (UniqueName: \"kubernetes.io/projected/3fc2cde5-218c-48bf-bad6-45ffd19da41b-kube-api-access-hsbg6\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:02.291421 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.291429    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc2cde5-218c-48bf-bad6-45ffd19da41b-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:02.902326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.902292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc" event={"ID":"3fc2cde5-218c-48bf-bad6-45ffd19da41b","Type":"ContainerDied","Data":"1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf"}
Jul 15 10:54:02.902326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.902328    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7gpc"
Jul 15 10:54:02.902326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:02.902338    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d7f2d6e413eef9cbd44a2ed9cb0652ed5eb6aaf9f96075bb647ddf61bbc3cdf"
Jul 15 10:54:05.639937 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:05.639900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:05.780472 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:05.780423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:14.813287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:14.813253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:19.727703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.727646    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cb99c7fd9-qqhvd" podUID="50649d4b-0874-4cac-b0f0-8760c4a9e090" containerName="console" containerID="cri-o://fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0" gracePeriod=15
Jul 15 10:54:19.976999 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.976975    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:54:19.981715 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.981658    2565 generic.go:356] "Generic (PLEG): container finished" podID="50649d4b-0874-4cac-b0f0-8760c4a9e090" containerID="fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0" exitCode=0
Jul 15 10:54:19.981715 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.981693    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99c7fd9-qqhvd"
Jul 15 10:54:19.981845 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.981737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99c7fd9-qqhvd" event={"ID":"50649d4b-0874-4cac-b0f0-8760c4a9e090","Type":"ContainerDied","Data":"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"}
Jul 15 10:54:19.981845 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.981790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99c7fd9-qqhvd" event={"ID":"50649d4b-0874-4cac-b0f0-8760c4a9e090","Type":"ContainerDied","Data":"18180017310c3cb806586c98f26b5a29e81f08d2744d3cbf71f57622f713a4e1"}
Jul 15 10:54:19.981845 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.981813    2565 scope.go:117] "RemoveContainer" containerID="fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"
Jul 15 10:54:19.989970 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.989953    2565 scope.go:117] "RemoveContainer" containerID="fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"
Jul 15 10:54:19.990257 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:54:19.990240    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0\": container with ID starting with fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0 not found: ID does not exist" containerID="fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"
Jul 15 10:54:19.990311 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:19.990265    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0"} err="failed to get container status \"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0\": rpc error: code = NotFound desc = could not find container \"fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0\": container with ID starting with fdb1522e8c6f91046f4466c3b1a8d73a6cabb1e6e77fb3b19b7282ec4fc4b7e0 not found: ID does not exist"
Jul 15 10:54:20.040232 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040195    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040385 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040385 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040297    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:20.040385 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040327    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040385 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040560 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kn4v\" (UniqueName: \"kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040560 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040560 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040442    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca\") pod \"50649d4b-0874-4cac-b0f0-8760c4a9e090\" (UID: \"50649d4b-0874-4cac-b0f0-8760c4a9e090\") "
Jul 15 10:54:20.040560 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:54:20.040768 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040708    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.040832 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040788    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config" (OuterVolumeSpecName: "console-config") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:54:20.040832 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:54:20.040967 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.040944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca" (OuterVolumeSpecName: "service-ca") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:54:20.042587 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.042559    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:54:20.042856 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.042802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:54:20.043001 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.042893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v" (OuterVolumeSpecName: "kube-api-access-9kn4v") pod "50649d4b-0874-4cac-b0f0-8760c4a9e090" (UID: "50649d4b-0874-4cac-b0f0-8760c4a9e090"). InnerVolumeSpecName "kube-api-access-9kn4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:20.052055 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052015    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"]
Jul 15 10:54:20.052390 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052372    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="pull"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052393    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="pull"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052418    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50649d4b-0874-4cac-b0f0-8760c4a9e090" containerName="console"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052424    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50649d4b-0874-4cac-b0f0-8760c4a9e090" containerName="console"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052442    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="extract"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052450    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="extract"
Jul 15 10:54:20.052473 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052471    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="util"
Jul 15 10:54:20.052787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052479    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="util"
Jul 15 10:54:20.052787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052568    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc2cde5-218c-48bf-bad6-45ffd19da41b" containerName="extract"
Jul 15 10:54:20.052787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.052585    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50649d4b-0874-4cac-b0f0-8760c4a9e090" containerName="console"
Jul 15 10:54:20.056985 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.056968    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.059426 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.059404    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 10:54:20.059536 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.059426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 15 10:54:20.059536 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.059468    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wd24x\""
Jul 15 10:54:20.059536 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.059408    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 15 10:54:20.059536 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.059409    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 15 10:54:20.067405 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.067385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"]
Jul 15 10:54:20.111291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.111257    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:20.142093 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjqr\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-kube-api-access-mmjqr\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142243 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142243 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142243 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142243 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d23-fa77-414b-be7f-0dea1442a91a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142352    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142368    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142383    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142397    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kn4v\" (UniqueName: \"kubernetes.io/projected/50649d4b-0874-4cac-b0f0-8760c4a9e090-kube-api-access-9kn4v\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142411    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.142459 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.142431    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50649d4b-0874-4cac-b0f0-8760c4a9e090-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:20.242945 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.242875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.242945 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.242919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d23-fa77-414b-be7f-0dea1442a91a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.242945 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.242934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.243231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.242973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjqr\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-kube-api-access-mmjqr\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.243231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.242994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.243231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.243020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.243231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.243034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.243819 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.243679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.245463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.245433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.245463 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.245458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.245580 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.245483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d23-fa77-414b-be7f-0dea1442a91a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.245580 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.245547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/435a8d23-fa77-414b-be7f-0dea1442a91a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.251353 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.251323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.251650 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.251632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjqr\" (UniqueName: \"kubernetes.io/projected/435a8d23-fa77-414b-be7f-0dea1442a91a-kube-api-access-mmjqr\") pod \"istiod-openshift-gateway-94bb8fbfd-mzhsm\" (UID: \"435a8d23-fa77-414b-be7f-0dea1442a91a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.303263 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.303239    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:54:20.310075 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.310055    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cb99c7fd9-qqhvd"]
Jul 15 10:54:20.366468 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.366447    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:20.501391 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.501365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"]
Jul 15 10:54:20.504103 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:54:20.504070    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435a8d23_fa77_414b_be7f_0dea1442a91a.slice/crio-d6288d78a9ca76c7abd62e6083ad9489c2da10235ed05600583fd7671fa525c1 WatchSource:0}: Error finding container d6288d78a9ca76c7abd62e6083ad9489c2da10235ed05600583fd7671fa525c1: Status 404 returned error can't find the container with id d6288d78a9ca76c7abd62e6083ad9489c2da10235ed05600583fd7671fa525c1
Jul 15 10:54:20.673758 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.673724    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50649d4b-0874-4cac-b0f0-8760c4a9e090" path="/var/lib/kubelet/pods/50649d4b-0874-4cac-b0f0-8760c4a9e090/volumes"
Jul 15 10:54:20.988815 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:20.988783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm" event={"ID":"435a8d23-fa77-414b-be7f-0dea1442a91a","Type":"ContainerStarted","Data":"d6288d78a9ca76c7abd62e6083ad9489c2da10235ed05600583fd7671fa525c1"}
Jul 15 10:54:21.148974 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:21.148939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:23.087613 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:23.087560    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 10:54:23.087971 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:23.087664    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 10:54:24.006660 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.006629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm" event={"ID":"435a8d23-fa77-414b-be7f-0dea1442a91a","Type":"ContainerStarted","Data":"7c19982cd24a766bc62438aa7ff2b3d4581d61716910571ff8442b14db438082"}
Jul 15 10:54:24.006857 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.006826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:24.008509 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.008484    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-mzhsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 10:54:24.008605 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.008537    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm" podUID="435a8d23-fa77-414b-be7f-0dea1442a91a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 10:54:24.137824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.137793    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:24.214622 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:24.214593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:25.011543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:25.011470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm"
Jul 15 10:54:25.034225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:25.034174    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm" podStartSLOduration=2.452800322 podStartE2EDuration="5.03415832s" totalImagesPullingTime="2.581357998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:54:24.036564042 +0000 UTC m=+529.871308492" watchObservedRunningTime="2026-07-15 10:54:25.03415832 +0000 UTC m=+530.868902770"
Jul 15 10:54:26.441388 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:26.441356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:26.519367 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:26.519334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:26.641711 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:26.641683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:30.065451 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:30.065426    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:31.119633 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:31.119601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:31.223740 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:31.223691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:32.113472 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:32.113438    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:32.211727 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:32.211696    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:35.197590 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.197554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"]
Jul 15 10:54:35.202430 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.202410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.205231 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.205211    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 15 10:54:35.205358 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.205244    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 15 10:54:35.205795 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.205779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bl6lr\""
Jul 15 10:54:35.213799 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.213776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"]
Jul 15 10:54:35.269615 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.269589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.269715 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.269648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.269756 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.269738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5887r\" (UniqueName: \"kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.292978 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.292954    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"]
Jul 15 10:54:35.295328 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.295312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.305145 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.305114    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"]
Jul 15 10:54:35.370331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.370462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.370462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.370462 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjvh\" (UniqueName: \"kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.370699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5887r\" (UniqueName: \"kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.370699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.370699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.370860 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.370827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.380604 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.380585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5887r\" (UniqueName: \"kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.392986 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.392963    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"]
Jul 15 10:54:35.395593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.395578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.404613 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.404590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"]
Jul 15 10:54:35.471204 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjvh\" (UniqueName: \"kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.471204 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.471204 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67rr\" (UniqueName: \"kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.471456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.471456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.471456 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.471601 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471580    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.471638 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.471612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.479823 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.479799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjvh\" (UniqueName: \"kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.493287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.493267    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"]
Jul 15 10:54:35.495987 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.495971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.505824 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.505802    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"]
Jul 15 10:54:35.511712 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.511692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:35.572119 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.571899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.572119 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.571959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.572119 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq484\" (UniqueName: \"kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.572366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.572366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.572366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67rr\" (UniqueName: \"kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.572366 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.572550 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.572430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.582074 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.582028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67rr\" (UniqueName: \"kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.604396 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.604367    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:35.650193 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.650171    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"]
Jul 15 10:54:35.655694 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:54:35.655664    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21b45c5_e407_4d31_abc0_800f19ad075a.slice/crio-1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5 WatchSource:0}: Error finding container 1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5: Status 404 returned error can't find the container with id 1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5
Jul 15 10:54:35.672932 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.672906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.673057 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.672978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq484\" (UniqueName: \"kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.673057 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.673027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.673344 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.673320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.673416 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.673367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.682814 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.682788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq484\" (UniqueName: \"kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.705265 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.705240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:35.749154 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.749069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"]
Jul 15 10:54:35.751852 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:54:35.751812    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1492d40c_b3d2_4880_93a0_d48a8c7213e0.slice/crio-d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09 WatchSource:0}: Error finding container d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09: Status 404 returned error can't find the container with id d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09
Jul 15 10:54:35.805545 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.805466    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:35.855709 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.855686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"]
Jul 15 10:54:35.857907 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:54:35.857884    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078dccec_95fd_4e3d_b5eb_8125b0dc1113.slice/crio-dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68 WatchSource:0}: Error finding container dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68: Status 404 returned error can't find the container with id dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68
Jul 15 10:54:35.955181 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:35.955150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"]
Jul 15 10:54:35.957926 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:54:35.957901    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb90b07_f97f_4e7c_b81e_13919f2d71c8.slice/crio-57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132 WatchSource:0}: Error finding container 57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132: Status 404 returned error can't find the container with id 57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132
Jul 15 10:54:36.063857 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.063823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr" event={"ID":"7eb90b07-f97f-4e7c-b81e-13919f2d71c8","Type":"ContainerStarted","Data":"57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132"}
Jul 15 10:54:36.065334 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.065304    2565 generic.go:356] "Generic (PLEG): container finished" podID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerID="17fbc0fe564c8fd158ac04943f91c73ea6d4eb2d515445965859392766bce874" exitCode=0
Jul 15 10:54:36.065429 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.065388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82" event={"ID":"078dccec-95fd-4e3d-b5eb-8125b0dc1113","Type":"ContainerDied","Data":"17fbc0fe564c8fd158ac04943f91c73ea6d4eb2d515445965859392766bce874"}
Jul 15 10:54:36.065488 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.065429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82" event={"ID":"078dccec-95fd-4e3d-b5eb-8125b0dc1113","Type":"ContainerStarted","Data":"dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68"}
Jul 15 10:54:36.066928 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.066903    2565 generic.go:356] "Generic (PLEG): container finished" podID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerID="fcbb5d51411b50d0a6424d16edd21fcb7ccfa6cfcd628f6d74e2147af32c00cf" exitCode=0
Jul 15 10:54:36.066999 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.066984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2" event={"ID":"1492d40c-b3d2-4880-93a0-d48a8c7213e0","Type":"ContainerDied","Data":"fcbb5d51411b50d0a6424d16edd21fcb7ccfa6cfcd628f6d74e2147af32c00cf"}
Jul 15 10:54:36.067078 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.067020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2" event={"ID":"1492d40c-b3d2-4880-93a0-d48a8c7213e0","Type":"ContainerStarted","Data":"d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09"}
Jul 15 10:54:36.068691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.068662    2565 generic.go:356] "Generic (PLEG): container finished" podID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerID="90705a6553ef219ed094e1a76847b4ef9395d041a04bd8a499bed186baf64b6c" exitCode=0
Jul 15 10:54:36.068798 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.068707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb" event={"ID":"a21b45c5-e407-4d31-abc0-800f19ad075a","Type":"ContainerDied","Data":"90705a6553ef219ed094e1a76847b4ef9395d041a04bd8a499bed186baf64b6c"}
Jul 15 10:54:36.068798 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:36.068754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb" event={"ID":"a21b45c5-e407-4d31-abc0-800f19ad075a","Type":"ContainerStarted","Data":"1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5"}
Jul 15 10:54:37.075709 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:37.075675    2565 generic.go:356] "Generic (PLEG): container finished" podID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerID="7f8d49b5b8636b9135e180593dbca10d11181b24c14d374c10b7f55d296ac893" exitCode=0
Jul 15 10:54:37.076123 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:37.075761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82" event={"ID":"078dccec-95fd-4e3d-b5eb-8125b0dc1113","Type":"ContainerDied","Data":"7f8d49b5b8636b9135e180593dbca10d11181b24c14d374c10b7f55d296ac893"}
Jul 15 10:54:37.077137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:37.077120    2565 generic.go:356] "Generic (PLEG): container finished" podID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerID="d27a4f5e0a8dcbbc5d5e9d85212d1dc2d1308cca0a13ecfff0361c116e180045" exitCode=0
Jul 15 10:54:37.077230 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:37.077168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr" event={"ID":"7eb90b07-f97f-4e7c-b81e-13919f2d71c8","Type":"ContainerDied","Data":"d27a4f5e0a8dcbbc5d5e9d85212d1dc2d1308cca0a13ecfff0361c116e180045"}
Jul 15 10:54:38.085251 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:38.085228    2565 generic.go:356] "Generic (PLEG): container finished" podID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerID="264a7aec7464c4a4b495cdd843ae28b9a0160863f4e07bb5c1941a766192736b" exitCode=0
Jul 15 10:54:38.085651 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:38.085308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82" event={"ID":"078dccec-95fd-4e3d-b5eb-8125b0dc1113","Type":"ContainerDied","Data":"264a7aec7464c4a4b495cdd843ae28b9a0160863f4e07bb5c1941a766192736b"}
Jul 15 10:54:38.087282 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:38.087259    2565 generic.go:356] "Generic (PLEG): container finished" podID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerID="80786d4f5d2bb71559bec0dcd3e8d5bc5d21c36d138988365f33a6bc5c78172d" exitCode=0
Jul 15 10:54:38.087365 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:38.087342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb" event={"ID":"a21b45c5-e407-4d31-abc0-800f19ad075a","Type":"ContainerDied","Data":"80786d4f5d2bb71559bec0dcd3e8d5bc5d21c36d138988365f33a6bc5c78172d"}
Jul 15 10:54:39.094214 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.094177    2565 generic.go:356] "Generic (PLEG): container finished" podID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerID="62ef7627f6d0ed2915d5c6613144022a056c38e542e7119a67a70ae57f1a613c" exitCode=0
Jul 15 10:54:39.094631 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.094268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb" event={"ID":"a21b45c5-e407-4d31-abc0-800f19ad075a","Type":"ContainerDied","Data":"62ef7627f6d0ed2915d5c6613144022a056c38e542e7119a67a70ae57f1a613c"}
Jul 15 10:54:39.096008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.095982    2565 generic.go:356] "Generic (PLEG): container finished" podID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerID="7473d3d3af6629026ac4d849f8d4d3bb0beabadd0f39357268865a08571f781a" exitCode=0
Jul 15 10:54:39.096165 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.096075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr" event={"ID":"7eb90b07-f97f-4e7c-b81e-13919f2d71c8","Type":"ContainerDied","Data":"7473d3d3af6629026ac4d849f8d4d3bb0beabadd0f39357268865a08571f781a"}
Jul 15 10:54:39.229121 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.229095    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:39.306789 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.306760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util\") pod \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") "
Jul 15 10:54:39.306907 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.306841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle\") pod \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") "
Jul 15 10:54:39.306907 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.306866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x67rr\" (UniqueName: \"kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr\") pod \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\" (UID: \"078dccec-95fd-4e3d-b5eb-8125b0dc1113\") "
Jul 15 10:54:39.307317 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.307286    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle" (OuterVolumeSpecName: "bundle") pod "078dccec-95fd-4e3d-b5eb-8125b0dc1113" (UID: "078dccec-95fd-4e3d-b5eb-8125b0dc1113"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:39.309038 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.309013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr" (OuterVolumeSpecName: "kube-api-access-x67rr") pod "078dccec-95fd-4e3d-b5eb-8125b0dc1113" (UID: "078dccec-95fd-4e3d-b5eb-8125b0dc1113"). InnerVolumeSpecName "kube-api-access-x67rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:39.312179 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.312156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util" (OuterVolumeSpecName: "util") pod "078dccec-95fd-4e3d-b5eb-8125b0dc1113" (UID: "078dccec-95fd-4e3d-b5eb-8125b0dc1113"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:39.408422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.408347    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:39.408422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.408373    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67rr\" (UniqueName: \"kubernetes.io/projected/078dccec-95fd-4e3d-b5eb-8125b0dc1113-kube-api-access-x67rr\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:39.408422 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:39.408382    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/078dccec-95fd-4e3d-b5eb-8125b0dc1113-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:40.103060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.103016    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82"
Jul 15 10:54:40.103060 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.103030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36xxt82" event={"ID":"078dccec-95fd-4e3d-b5eb-8125b0dc1113","Type":"ContainerDied","Data":"dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68"}
Jul 15 10:54:40.103505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.103092    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dada375ba092d786bf8142a729a0ba69bc39589a5fd88a8f0ba67b950f08dc68"
Jul 15 10:54:40.105190 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.105160    2565 generic.go:356] "Generic (PLEG): container finished" podID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerID="1cf1a732b6cff6924323ab0411654bc516c8303aa5db5aef1d3d7840c172a2e8" exitCode=0
Jul 15 10:54:40.105323 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.105238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr" event={"ID":"7eb90b07-f97f-4e7c-b81e-13919f2d71c8","Type":"ContainerDied","Data":"1cf1a732b6cff6924323ab0411654bc516c8303aa5db5aef1d3d7840c172a2e8"}
Jul 15 10:54:40.249747 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.249721    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:40.314547 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.314524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle\") pod \"a21b45c5-e407-4d31-abc0-800f19ad075a\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") "
Jul 15 10:54:40.314699 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.314567    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util\") pod \"a21b45c5-e407-4d31-abc0-800f19ad075a\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") "
Jul 15 10:54:40.314760 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.314741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5887r\" (UniqueName: \"kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r\") pod \"a21b45c5-e407-4d31-abc0-800f19ad075a\" (UID: \"a21b45c5-e407-4d31-abc0-800f19ad075a\") "
Jul 15 10:54:40.315006 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.314977    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle" (OuterVolumeSpecName: "bundle") pod "a21b45c5-e407-4d31-abc0-800f19ad075a" (UID: "a21b45c5-e407-4d31-abc0-800f19ad075a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:40.316937 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.316918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r" (OuterVolumeSpecName: "kube-api-access-5887r") pod "a21b45c5-e407-4d31-abc0-800f19ad075a" (UID: "a21b45c5-e407-4d31-abc0-800f19ad075a"). InnerVolumeSpecName "kube-api-access-5887r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:40.320450 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.320419    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util" (OuterVolumeSpecName: "util") pod "a21b45c5-e407-4d31-abc0-800f19ad075a" (UID: "a21b45c5-e407-4d31-abc0-800f19ad075a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:40.415508 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.415442    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5887r\" (UniqueName: \"kubernetes.io/projected/a21b45c5-e407-4d31-abc0-800f19ad075a-kube-api-access-5887r\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:40.415508 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.415469    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:40.415508 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:40.415478    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a21b45c5-e407-4d31-abc0-800f19ad075a-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:41.111848 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.111822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb"
Jul 15 10:54:41.112219 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.111842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317ef7j7cb" event={"ID":"a21b45c5-e407-4d31-abc0-800f19ad075a","Type":"ContainerDied","Data":"1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5"}
Jul 15 10:54:41.112219 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.111888    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cfb1adcf2309295351b0952f99de4a9468a3d6a2561be91d914e575960e99d5"
Jul 15 10:54:41.248137 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.248117    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:41.322648 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.322624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq484\" (UniqueName: \"kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484\") pod \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") "
Jul 15 10:54:41.322648 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.322654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util\") pod \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") "
Jul 15 10:54:41.322810 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.322724    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle\") pod \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\" (UID: \"7eb90b07-f97f-4e7c-b81e-13919f2d71c8\") "
Jul 15 10:54:41.323257 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.323230    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle" (OuterVolumeSpecName: "bundle") pod "7eb90b07-f97f-4e7c-b81e-13919f2d71c8" (UID: "7eb90b07-f97f-4e7c-b81e-13919f2d71c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:41.324787 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.324764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484" (OuterVolumeSpecName: "kube-api-access-tq484") pod "7eb90b07-f97f-4e7c-b81e-13919f2d71c8" (UID: "7eb90b07-f97f-4e7c-b81e-13919f2d71c8"). InnerVolumeSpecName "kube-api-access-tq484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:41.350331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.350293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util" (OuterVolumeSpecName: "util") pod "7eb90b07-f97f-4e7c-b81e-13919f2d71c8" (UID: "7eb90b07-f97f-4e7c-b81e-13919f2d71c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:41.423303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.423245    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:41.423303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.423268    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq484\" (UniqueName: \"kubernetes.io/projected/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-kube-api-access-tq484\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:41.423303 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:41.423277    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb90b07-f97f-4e7c-b81e-13919f2d71c8-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:42.118961 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:42.118919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr" event={"ID":"7eb90b07-f97f-4e7c-b81e-13919f2d71c8","Type":"ContainerDied","Data":"57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132"}
Jul 15 10:54:42.118961 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:42.118965    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabts8fr"
Jul 15 10:54:42.119386 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:42.119008    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d17497912611cb56cb6c26bb0b8e3f65d3a35127e5ca09610f3856c7d25132"
Jul 15 10:54:54.117218 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:54.117179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:54.176323 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:54.176275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:54:56.188736 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:56.188699    2565 generic.go:356] "Generic (PLEG): container finished" podID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerID="660d8a9089750a873c5e9f15cf15a0e3703685291ac63b11fd2b55421c37569c" exitCode=0
Jul 15 10:54:56.189142 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:56.188769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2" event={"ID":"1492d40c-b3d2-4880-93a0-d48a8c7213e0","Type":"ContainerDied","Data":"660d8a9089750a873c5e9f15cf15a0e3703685291ac63b11fd2b55421c37569c"}
Jul 15 10:54:57.195969 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:57.195934    2565 generic.go:356] "Generic (PLEG): container finished" podID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerID="a763d46532b6f2054e1b9fd70c909d7b300f9722ec177ede32de47bab47ccce6" exitCode=0
Jul 15 10:54:57.196388 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:57.195989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2" event={"ID":"1492d40c-b3d2-4880-93a0-d48a8c7213e0","Type":"ContainerDied","Data":"a763d46532b6f2054e1b9fd70c909d7b300f9722ec177ede32de47bab47ccce6"}
Jul 15 10:54:58.333230 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.333204    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:58.475376 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.475293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kjvh\" (UniqueName: \"kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh\") pod \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") "
Jul 15 10:54:58.475376 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.475355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle\") pod \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") "
Jul 15 10:54:58.475593 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.475385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util\") pod \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\" (UID: \"1492d40c-b3d2-4880-93a0-d48a8c7213e0\") "
Jul 15 10:54:58.476008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.475983    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle" (OuterVolumeSpecName: "bundle") pod "1492d40c-b3d2-4880-93a0-d48a8c7213e0" (UID: "1492d40c-b3d2-4880-93a0-d48a8c7213e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:58.477612 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.477589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh" (OuterVolumeSpecName: "kube-api-access-8kjvh") pod "1492d40c-b3d2-4880-93a0-d48a8c7213e0" (UID: "1492d40c-b3d2-4880-93a0-d48a8c7213e0"). InnerVolumeSpecName "kube-api-access-8kjvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:54:58.480080 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.480057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util" (OuterVolumeSpecName: "util") pod "1492d40c-b3d2-4880-93a0-d48a8c7213e0" (UID: "1492d40c-b3d2-4880-93a0-d48a8c7213e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:54:58.577000 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.576965    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjvh\" (UniqueName: \"kubernetes.io/projected/1492d40c-b3d2-4880-93a0-d48a8c7213e0-kube-api-access-8kjvh\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:58.577000 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.576994    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:58.577000 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:58.577008    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1492d40c-b3d2-4880-93a0-d48a8c7213e0-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:54:59.208116 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:59.208078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2" event={"ID":"1492d40c-b3d2-4880-93a0-d48a8c7213e0","Type":"ContainerDied","Data":"d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09"}
Jul 15 10:54:59.208116 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:59.208107    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hcnq2"
Jul 15 10:54:59.208330 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:54:59.208135    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d77f106a3af1298242feaf41cc028f98a974ea2dacbb2f1726a837154e248d09"
Jul 15 10:55:06.461491 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:06.461423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:10.594701 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:10.594667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:13.667421 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667391    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"]
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667718    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="pull"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667729    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="pull"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="pull"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667748    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="pull"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="util"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="util"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="extract"
Jul 15 10:55:13.667775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667778    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667790    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667800    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667805    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667813    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667817    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667826    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667831    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="util"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667842    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="pull"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="pull"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667852    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="pull"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="pull"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667865    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667870    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667917    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="078dccec-95fd-4e3d-b5eb-8125b0dc1113" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667926    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21b45c5-e407-4d31-abc0-800f19ad075a" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667935    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1492d40c-b3d2-4880-93a0-d48a8c7213e0" containerName="extract"
Jul 15 10:55:13.668023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.667940    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb90b07-f97f-4e7c-b81e-13919f2d71c8" containerName="extract"
Jul 15 10:55:13.672364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.672339    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:13.675026 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.675007    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 15 10:55:13.675873 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.675855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 15 10:55:13.675954 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.675858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7vvnk\""
Jul 15 10:55:13.683742 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.683721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"]
Jul 15 10:55:13.698908 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.698884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8lf\" (UniqueName: \"kubernetes.io/projected/b7de55b9-da42-4bc9-be97-975ea925f74e-kube-api-access-st8lf\") pod \"limitador-operator-controller-manager-6f9f468797-r4hbr\" (UID: \"b7de55b9-da42-4bc9-be97-975ea925f74e\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:13.800397 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.800363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8lf\" (UniqueName: \"kubernetes.io/projected/b7de55b9-da42-4bc9-be97-975ea925f74e-kube-api-access-st8lf\") pod \"limitador-operator-controller-manager-6f9f468797-r4hbr\" (UID: \"b7de55b9-da42-4bc9-be97-975ea925f74e\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:13.817731 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.817698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8lf\" (UniqueName: \"kubernetes.io/projected/b7de55b9-da42-4bc9-be97-975ea925f74e-kube-api-access-st8lf\") pod \"limitador-operator-controller-manager-6f9f468797-r4hbr\" (UID: \"b7de55b9-da42-4bc9-be97-975ea925f74e\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:13.982182 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:13.982095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:14.126515 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:14.126492    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"]
Jul 15 10:55:14.128448 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:55:14.128420    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7de55b9_da42_4bc9_be97_975ea925f74e.slice/crio-84ee3805caf25017618469539afda8193f86dbc6c0082b199b6638813337506c WatchSource:0}: Error finding container 84ee3805caf25017618469539afda8193f86dbc6c0082b199b6638813337506c: Status 404 returned error can't find the container with id 84ee3805caf25017618469539afda8193f86dbc6c0082b199b6638813337506c
Jul 15 10:55:14.283627 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:14.283539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr" event={"ID":"b7de55b9-da42-4bc9-be97-975ea925f74e","Type":"ContainerStarted","Data":"84ee3805caf25017618469539afda8193f86dbc6c0082b199b6638813337506c"}
Jul 15 10:55:14.895455 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:14.895424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:15.006678 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:15.006639    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:15.804162 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:15.804131    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:16.296543 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:16.296505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr" event={"ID":"b7de55b9-da42-4bc9-be97-975ea925f74e","Type":"ContainerStarted","Data":"8fcacbcd9a4a68910467c118282a299b89dfe721abab62b67d40a7200819f725"}
Jul 15 10:55:16.296965 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:16.296661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:16.316331 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:16.316287    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr" podStartSLOduration=1.818921555 podStartE2EDuration="3.316273834s" totalImagesPullingTime="1.497352279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:16.314580705 +0000 UTC m=+582.149325181" watchObservedRunningTime="2026-07-15 10:55:16.316273834 +0000 UTC m=+582.151018283"
Jul 15 10:55:20.398784 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:20.398749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:20.981976 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:20.981948    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:22.675544 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:22.675512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:23.283163 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.283129    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:23.378515 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.378481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:23.690912 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.690867    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-45x42"]
Jul 15 10:55:23.694414 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.694395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:23.696881 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.696862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pqf69\""
Jul 15 10:55:23.707299 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.707278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-45x42"]
Jul 15 10:55:23.783319 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.783289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dpw\" (UniqueName: \"kubernetes.io/projected/2a1022a7-a38b-422a-a575-7d634cb767a8-kube-api-access-c5dpw\") pod \"authorino-operator-6d85f6564-45x42\" (UID: \"2a1022a7-a38b-422a-a575-7d634cb767a8\") " pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:23.883736 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.883707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dpw\" (UniqueName: \"kubernetes.io/projected/2a1022a7-a38b-422a-a575-7d634cb767a8-kube-api-access-c5dpw\") pod \"authorino-operator-6d85f6564-45x42\" (UID: \"2a1022a7-a38b-422a-a575-7d634cb767a8\") " pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:23.894958 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:23.894933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dpw\" (UniqueName: \"kubernetes.io/projected/2a1022a7-a38b-422a-a575-7d634cb767a8-kube-api-access-c5dpw\") pod \"authorino-operator-6d85f6564-45x42\" (UID: \"2a1022a7-a38b-422a-a575-7d634cb767a8\") " pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:24.004337 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:24.004243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:24.155414 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:24.155390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-45x42"]
Jul 15 10:55:24.157537 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:55:24.157513    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1022a7_a38b_422a_a575_7d634cb767a8.slice/crio-33736b3ea6dc69f892935d5b890487b52ff35f7d0f86c7f7c00471b7e4626e1a WatchSource:0}: Error finding container 33736b3ea6dc69f892935d5b890487b52ff35f7d0f86c7f7c00471b7e4626e1a: Status 404 returned error can't find the container with id 33736b3ea6dc69f892935d5b890487b52ff35f7d0f86c7f7c00471b7e4626e1a
Jul 15 10:55:24.336298 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:24.336271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-45x42" event={"ID":"2a1022a7-a38b-422a-a575-7d634cb767a8","Type":"ContainerStarted","Data":"33736b3ea6dc69f892935d5b890487b52ff35f7d0f86c7f7c00471b7e4626e1a"}
Jul 15 10:55:24.797644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:24.797561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:25.057970 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:25.057883    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:26.532926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.532902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:26.629559 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.629523    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"]
Jul 15 10:55:26.632996 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.632974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:26.636498 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.636482    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-phrcw\""
Jul 15 10:55:26.636635 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.636620    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 15 10:55:26.647906 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.647884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"]
Jul 15 10:55:26.708066 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.708012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckk8t\" (UniqueName: \"kubernetes.io/projected/4d35bccc-aa92-4731-9f68-4ece273ee422-kube-api-access-ckk8t\") pod \"dns-operator-controller-manager-65b49595d7-gc6m6\" (UID: \"4d35bccc-aa92-4731-9f68-4ece273ee422\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:26.808979 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.808896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckk8t\" (UniqueName: \"kubernetes.io/projected/4d35bccc-aa92-4731-9f68-4ece273ee422-kube-api-access-ckk8t\") pod \"dns-operator-controller-manager-65b49595d7-gc6m6\" (UID: \"4d35bccc-aa92-4731-9f68-4ece273ee422\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:26.817585 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.817564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckk8t\" (UniqueName: \"kubernetes.io/projected/4d35bccc-aa92-4731-9f68-4ece273ee422-kube-api-access-ckk8t\") pod \"dns-operator-controller-manager-65b49595d7-gc6m6\" (UID: \"4d35bccc-aa92-4731-9f68-4ece273ee422\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:26.943411 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:26.943383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:27.079480 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.079450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"]
Jul 15 10:55:27.083234 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:55:27.083208    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d35bccc_aa92_4731_9f68_4ece273ee422.slice/crio-18136106027db67687be5db5b5b4a83be1b40edb22b65786ba77d19d0d3d0ea8 WatchSource:0}: Error finding container 18136106027db67687be5db5b5b4a83be1b40edb22b65786ba77d19d0d3d0ea8: Status 404 returned error can't find the container with id 18136106027db67687be5db5b5b4a83be1b40edb22b65786ba77d19d0d3d0ea8
Jul 15 10:55:27.303583 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.303552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr"
Jul 15 10:55:27.355598 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.355502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-45x42" event={"ID":"2a1022a7-a38b-422a-a575-7d634cb767a8","Type":"ContainerStarted","Data":"d2880a77825e468626b0cf426fdf6dbbbc64abfcd67df67327bd07f62cd1a698"}
Jul 15 10:55:27.355747 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.355627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:27.357090 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.357036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6" event={"ID":"4d35bccc-aa92-4731-9f68-4ece273ee422","Type":"ContainerStarted","Data":"18136106027db67687be5db5b5b4a83be1b40edb22b65786ba77d19d0d3d0ea8"}
Jul 15 10:55:27.382024 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.381976    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-45x42" podStartSLOduration=2.056481805 podStartE2EDuration="4.38196174s" totalImagesPullingTime="2.325479935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:27.380504381 +0000 UTC m=+593.215248865" watchObservedRunningTime="2026-07-15 10:55:27.38196174 +0000 UTC m=+593.216706189"
Jul 15 10:55:27.458450 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.458413    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:27.594939 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.594901    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:27.798161 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:27.798066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:28.047450 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:28.047419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:28.145413 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:28.145372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:29.370247 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:29.370213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6" event={"ID":"4d35bccc-aa92-4731-9f68-4ece273ee422","Type":"ContainerStarted","Data":"e68db468d34bd25c22cde15738163b6f610ec15a001251f6fd10dacec6762bf5"}
Jul 15 10:55:29.370641 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:29.370359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:29.391155 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:29.391099    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6" podStartSLOduration=1.871793608 podStartE2EDuration="3.391084096s" totalImagesPullingTime="1.519290488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:55:29.389651218 +0000 UTC m=+595.224395669" watchObservedRunningTime="2026-07-15 10:55:29.391084096 +0000 UTC m=+595.225828558"
Jul 15 10:55:31.988340 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:31.988309    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:32.089353 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:32.089326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:32.399549 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:32.399521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:32.463600 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:32.463565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:32.790706 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:32.790632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:32.866284 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:32.866250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:34.572148 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:34.572110    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:55:34.574615 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:34.574593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 10:55:34.576136 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:34.576106    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:55:34.578434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:34.578413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 10:55:38.366572 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:38.366539    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-45x42"
Jul 15 10:55:40.378283 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:40.378252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6"
Jul 15 10:55:40.493496 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:40.493464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:40.591385 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:40.591355    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:42.934659 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:42.934625    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:43.032287 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:43.032256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:47.873864 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:47.873826    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:47.931443 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:47.931411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:50.169884 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:50.169849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:50.270310 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:50.270278    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:51.942418 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:51.942382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:52.000466 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:52.000435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:52.089008 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:52.088976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:52.224215 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:52.224133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:53.089635 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:53.089600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:55:53.150969 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:55:53.150939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:01.495644 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:01.495615    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:01.696394 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:01.696359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:01.792063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:01.791959    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:03.971580 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:03.971543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:04.039832 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:04.039801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:10.506338 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:10.506305    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:11.492196 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.492162    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:11.494511 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.494494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:11.497596 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.497575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jcfqf\""
Jul 15 10:56:11.504916 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.504895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:11.580400 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.580353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcnb\" (UniqueName: \"kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb\") pod \"authorino-8df697c9f-md95b\" (UID: \"f5f24267-20f6-4a5e-9708-8cb6c8748f2e\") " pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:11.681606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.681569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcnb\" (UniqueName: \"kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb\") pod \"authorino-8df697c9f-md95b\" (UID: \"f5f24267-20f6-4a5e-9708-8cb6c8748f2e\") " pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:11.685217 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.685191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:11.691434 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.691413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcnb\" (UniqueName: \"kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb\") pod \"authorino-8df697c9f-md95b\" (UID: \"f5f24267-20f6-4a5e-9708-8cb6c8748f2e\") " pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:11.814998 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.814917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:11.946991 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.946963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:11.948904 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:56:11.948875    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f24267_20f6_4a5e_9708_8cb6c8748f2e.slice/crio-af739b0118148ed46c21db6553ab5757acd3fbb23e3b9ffe6ed2a6aac3e2a88b WatchSource:0}: Error finding container af739b0118148ed46c21db6553ab5757acd3fbb23e3b9ffe6ed2a6aac3e2a88b: Status 404 returned error can't find the container with id af739b0118148ed46c21db6553ab5757acd3fbb23e3b9ffe6ed2a6aac3e2a88b
Jul 15 10:56:11.950649 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:11.950634    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:56:12.599499 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:12.599468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-md95b" event={"ID":"f5f24267-20f6-4a5e-9708-8cb6c8748f2e","Type":"ContainerStarted","Data":"af739b0118148ed46c21db6553ab5757acd3fbb23e3b9ffe6ed2a6aac3e2a88b"}
Jul 15 10:56:21.109688 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:21.109657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:21.169810 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:21.169778    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:23.659708 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:23.659671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-md95b" event={"ID":"f5f24267-20f6-4a5e-9708-8cb6c8748f2e","Type":"ContainerStarted","Data":"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"}
Jul 15 10:56:23.677798 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:23.677740    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-md95b" podStartSLOduration=1.195174833 podStartE2EDuration="12.677721649s" totalImagesPullingTime="11.482546816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:23.675979321 +0000 UTC m=+649.510723750" watchObservedRunningTime="2026-07-15 10:56:23.677721649 +0000 UTC m=+649.512466101"
Jul 15 10:56:32.384290 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:32.384253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:33.088900 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:33.088862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:56:37.800780 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:37.800690    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:37.801291 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:37.800907    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-md95b" podUID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" containerName="authorino" containerID="cri-o://e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6" gracePeriod=30
Jul 15 10:56:38.050944 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.050883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:38.109662 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.109630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcnb\" (UniqueName: \"kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb\") pod \"f5f24267-20f6-4a5e-9708-8cb6c8748f2e\" (UID: \"f5f24267-20f6-4a5e-9708-8cb6c8748f2e\") "
Jul 15 10:56:38.111774 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.111748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb" (OuterVolumeSpecName: "kube-api-access-qfcnb") pod "f5f24267-20f6-4a5e-9708-8cb6c8748f2e" (UID: "f5f24267-20f6-4a5e-9708-8cb6c8748f2e"). InnerVolumeSpecName "kube-api-access-qfcnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:38.210486 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.210459    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcnb\" (UniqueName: \"kubernetes.io/projected/f5f24267-20f6-4a5e-9708-8cb6c8748f2e-kube-api-access-qfcnb\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:56:38.734271 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.734231    2565 generic.go:356] "Generic (PLEG): container finished" podID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" containerID="e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6" exitCode=0
Jul 15 10:56:38.734271 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.734269    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-md95b"
Jul 15 10:56:38.734469 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.734275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-md95b" event={"ID":"f5f24267-20f6-4a5e-9708-8cb6c8748f2e","Type":"ContainerDied","Data":"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"}
Jul 15 10:56:38.734469 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.734316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-md95b" event={"ID":"f5f24267-20f6-4a5e-9708-8cb6c8748f2e","Type":"ContainerDied","Data":"af739b0118148ed46c21db6553ab5757acd3fbb23e3b9ffe6ed2a6aac3e2a88b"}
Jul 15 10:56:38.734469 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.734340    2565 scope.go:117] "RemoveContainer" containerID="e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"
Jul 15 10:56:38.743225 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.743209    2565 scope.go:117] "RemoveContainer" containerID="e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"
Jul 15 10:56:38.743491 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:56:38.743474    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6\": container with ID starting with e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6 not found: ID does not exist" containerID="e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"
Jul 15 10:56:38.743549 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.743500    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6"} err="failed to get container status \"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6\": rpc error: code = NotFound desc = could not find container \"e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6\": container with ID starting with e98a00ed3369fc90cc0db717c1d05976ff32b196096f034a94f14aaeab9ba8e6 not found: ID does not exist"
Jul 15 10:56:38.758914 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.758891    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:38.762841 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:38.762821    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-md95b"]
Jul 15 10:56:40.673904 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:56:40.673876    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" path="/var/lib/kubelet/pods/f5f24267-20f6-4a5e-9708-8cb6c8748f2e/volumes"
Jul 15 10:57:15.807966 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.807935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:15.849028 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.848992    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-vbjvn"]
Jul 15 10:57:15.849381 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.849367    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" containerName="authorino"
Jul 15 10:57:15.849432 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.849382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" containerName="authorino"
Jul 15 10:57:15.849468 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.849452    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f24267-20f6-4a5e-9708-8cb6c8748f2e" containerName="authorino"
Jul 15 10:57:15.852485 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.852460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:15.855007 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.854985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 15 10:57:15.855152 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.855084    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8dhtf\""
Jul 15 10:57:15.868142 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.868112    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nfnsm"]
Jul 15 10:57:15.871696 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.871674    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vbjvn"]
Jul 15 10:57:15.871812 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.871769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:15.874334 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.874315    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 15 10:57:15.874542 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.874511    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-cg5f6\""
Jul 15 10:57:15.886101 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.886072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nfnsm"]
Jul 15 10:57:15.908267 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.908230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:15.927197 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.927169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:15.927351 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.927209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg29h\" (UniqueName: \"kubernetes.io/projected/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-kube-api-access-cg29h\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:15.927351 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.927242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvkk\" (UniqueName: \"kubernetes.io/projected/75601234-a086-4aeb-b916-b6e08a48c392-kube-api-access-rkvkk\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:15.927351 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.927291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-cert\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:15.972909 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:15.972874    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:16.028467 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.028431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-cert\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:16.028643 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.028499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.028643 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.028541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg29h\" (UniqueName: \"kubernetes.io/projected/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-kube-api-access-cg29h\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:16.028643 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.028577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvkk\" (UniqueName: \"kubernetes.io/projected/75601234-a086-4aeb-b916-b6e08a48c392-kube-api-access-rkvkk\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.028754 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:57:16.028653    2565 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 15 10:57:16.028754 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:57:16.028731    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs podName:75601234-a086-4aeb-b916-b6e08a48c392 nodeName:}" failed. No retries permitted until 2026-07-15 10:57:16.528714858 +0000 UTC m=+702.363459288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs") pod "model-serving-api-fd65d7d6-vbjvn" (UID: "75601234-a086-4aeb-b916-b6e08a48c392") : secret "model-serving-api-tls" not found
Jul 15 10:57:16.031113 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.031087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-cert\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:16.037240 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.037209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvkk\" (UniqueName: \"kubernetes.io/projected/75601234-a086-4aeb-b916-b6e08a48c392-kube-api-access-rkvkk\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.037387 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.037364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg29h\" (UniqueName: \"kubernetes.io/projected/4a3bd5d1-7c26-47d0-b797-0c3bc39987ed-kube-api-access-cg29h\") pod \"odh-model-controller-84b54b7f97-nfnsm\" (UID: \"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed\") " pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:16.101398 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.101369    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:16.182429 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.182394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:16.334107 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.330025    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nfnsm"]
Jul 15 10:57:16.336966 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:57:16.336936    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3bd5d1_7c26_47d0_b797_0c3bc39987ed.slice/crio-e60edf09a83e225e85351661c7e03e356c54c7ad8b474ef640ea3f526054dd07 WatchSource:0}: Error finding container e60edf09a83e225e85351661c7e03e356c54c7ad8b474ef640ea3f526054dd07: Status 404 returned error can't find the container with id e60edf09a83e225e85351661c7e03e356c54c7ad8b474ef640ea3f526054dd07
Jul 15 10:57:16.515312 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.515230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:16.537186 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.537147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.539635 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.539606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75601234-a086-4aeb-b916-b6e08a48c392-tls-certs\") pod \"model-serving-api-fd65d7d6-vbjvn\" (UID: \"75601234-a086-4aeb-b916-b6e08a48c392\") " pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.648911 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.648879    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:57:16.762943 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.762902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:16.925479 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.925444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vbjvn"]
Jul 15 10:57:16.927773 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:57:16.927736    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75601234_a086_4aeb_b916_b6e08a48c392.slice/crio-a05c7820a373a6d56dff09592ed495cd6fd7205ed4c214b1deacf0611ccc3e5b WatchSource:0}: Error finding container a05c7820a373a6d56dff09592ed495cd6fd7205ed4c214b1deacf0611ccc3e5b: Status 404 returned error can't find the container with id a05c7820a373a6d56dff09592ed495cd6fd7205ed4c214b1deacf0611ccc3e5b
Jul 15 10:57:16.936914 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:16.936873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-nfnsm" event={"ID":"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed","Type":"ContainerStarted","Data":"e60edf09a83e225e85351661c7e03e356c54c7ad8b474ef640ea3f526054dd07"}
Jul 15 10:57:17.943508 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:17.943469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vbjvn" event={"ID":"75601234-a086-4aeb-b916-b6e08a48c392","Type":"ContainerStarted","Data":"a05c7820a373a6d56dff09592ed495cd6fd7205ed4c214b1deacf0611ccc3e5b"}
Jul 15 10:57:19.959790 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.959751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vbjvn" event={"ID":"75601234-a086-4aeb-b916-b6e08a48c392","Type":"ContainerStarted","Data":"651b58f90bf7eb385f40c18af157e8e3fba471d9417c95e4843120b886a558a5"}
Jul 15 10:57:19.960312 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.959828    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:57:19.961381 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.961356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-nfnsm" event={"ID":"4a3bd5d1-7c26-47d0-b797-0c3bc39987ed","Type":"ContainerStarted","Data":"bf11cada1c7b445fcb75517b53739b88c3f7abd54bf169ce9b00cd1e68a69940"}
Jul 15 10:57:19.961523 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.961491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:19.977418 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.977373    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-vbjvn" podStartSLOduration=2.59190883 podStartE2EDuration="4.977340641s" totalImagesPullingTime="2.385431811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:19.97595618 +0000 UTC m=+705.810700629" watchObservedRunningTime="2026-07-15 10:57:19.977340641 +0000 UTC m=+705.812085102"
Jul 15 10:57:19.993058 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:19.992996    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-nfnsm" podStartSLOduration=2.011952906 podStartE2EDuration="4.992980897s" totalImagesPullingTime="2.981027991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:19.992030939 +0000 UTC m=+705.826775409" watchObservedRunningTime="2026-07-15 10:57:19.992980897 +0000 UTC m=+705.827725346"
Jul 15 10:57:30.968588 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:30.968508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-nfnsm"
Jul 15 10:57:30.970306 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:57:30.970282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-vbjvn"
Jul 15 10:58:08.717294 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:58:08.717254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:58:49.723301 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:58:49.723263    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:58:49.821098 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:58:49.821060    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:03.099701 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.099660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:03.182713 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.182666    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:03.257324 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.257287    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:03.260926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.260901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.263365 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.263340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 10:59:03.263491 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.263466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-95m52\""
Jul 15 10:59:03.263568 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.263556    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 10:59:03.263625 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.263556    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 10:59:03.273426 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.273402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:03.375311 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375483 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375656 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bph\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375732 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375795 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.375853 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.375835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477293 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477505 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bph\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.477919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.477789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.478744 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.478567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.478873 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.478796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.478947 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.478895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.487373 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.487298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.487735 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.487692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.490107 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.490082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.490919 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.490898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bph\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-shbsr\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.572642 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.572596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:03.725766 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:03.725741    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:03.727962 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:59:03.727928    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe7ba8f_198a_45bd_b973_9c806e2c3d65.slice/crio-73e8078d3df8edaadbd06c892ec047fff196a0fac6edd0e1662f09c0b3addbc7 WatchSource:0}: Error finding container 73e8078d3df8edaadbd06c892ec047fff196a0fac6edd0e1662f09c0b3addbc7: Status 404 returned error can't find the container with id 73e8078d3df8edaadbd06c892ec047fff196a0fac6edd0e1662f09c0b3addbc7
Jul 15 10:59:04.518775 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:04.518735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" event={"ID":"afe7ba8f-198a-45bd-b973-9c806e2c3d65","Type":"ContainerStarted","Data":"73e8078d3df8edaadbd06c892ec047fff196a0fac6edd0e1662f09c0b3addbc7"}
Jul 15 10:59:06.651887 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:06.651843    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 10:59:06.652233 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:06.651936    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 10:59:06.652233 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:06.651971    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 10:59:06.703322 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:06.703275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:07.538923 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:07.538884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" event={"ID":"afe7ba8f-198a-45bd-b973-9c806e2c3d65","Type":"ContainerStarted","Data":"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"}
Jul 15 10:59:07.560867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:07.560820    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" podStartSLOduration=1.639118858 podStartE2EDuration="4.560802914s" totalImagesPullingTime="2.921684056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:07.559502169 +0000 UTC m=+813.394246642" watchObservedRunningTime="2026-07-15 10:59:07.560802914 +0000 UTC m=+813.395547364"
Jul 15 10:59:07.575867 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:07.575814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:07.580743 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:07.580717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:08.544391 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:08.544355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:08.545500 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:08.545478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:18.892792 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:18.892757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:23.536842 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:23.536804    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:23.537452 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:23.537016    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerName="istio-proxy" containerID="cri-o://c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650" gracePeriod=30
Jul 15 10:59:23.554292 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:23.554248    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 10:59:29.099239 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.099180    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:29.218674 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7bph\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218674 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218674 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.218950 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.218911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token\") pod \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\" (UID: \"afe7ba8f-198a-45bd-b973-9c806e2c3d65\") "
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219233    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data" (OuterVolumeSpecName: "istio-data") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219253    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-credential-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219272    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.219326 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219287    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-workload-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.219862 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.219434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:59:29.221238 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.221215    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph" (OuterVolumeSpecName: "kube-api-access-f7bph") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "kube-api-access-f7bph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:29.221377 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.221218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:29.221659 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.221592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token" (OuterVolumeSpecName: "istio-token") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:29.221659 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.221615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afe7ba8f-198a-45bd-b973-9c806e2c3d65" (UID: "afe7ba8f-198a-45bd-b973-9c806e2c3d65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 10:59:29.320140 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320107    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-podinfo\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.320140 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320135    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istiod-ca-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.320140 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320144    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.320370 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320152    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bph\" (UniqueName: \"kubernetes.io/projected/afe7ba8f-198a-45bd-b973-9c806e2c3d65-kube-api-access-f7bph\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.320370 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320161    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-envoy\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.320370 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.320169    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe7ba8f-198a-45bd-b973-9c806e2c3d65-istio-data\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 10:59:29.659866 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.659834    2565 generic.go:356] "Generic (PLEG): container finished" podID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerID="c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650" exitCode=0
Jul 15 10:59:29.660063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.659882    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"
Jul 15 10:59:29.660063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.659913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" event={"ID":"afe7ba8f-198a-45bd-b973-9c806e2c3d65","Type":"ContainerDied","Data":"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"}
Jul 15 10:59:29.660063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.659968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr" event={"ID":"afe7ba8f-198a-45bd-b973-9c806e2c3d65","Type":"ContainerDied","Data":"73e8078d3df8edaadbd06c892ec047fff196a0fac6edd0e1662f09c0b3addbc7"}
Jul 15 10:59:29.660063 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.659992    2565 scope.go:117] "RemoveContainer" containerID="c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"
Jul 15 10:59:29.669815 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.669798    2565 scope.go:117] "RemoveContainer" containerID="c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"
Jul 15 10:59:29.670212 ip-10-0-137-92 kubenswrapper[2565]: E0715 10:59:29.670192    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650\": container with ID starting with c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650 not found: ID does not exist" containerID="c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"
Jul 15 10:59:29.670264 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.670221    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650"} err="failed to get container status \"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650\": rpc error: code = NotFound desc = could not find container \"c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650\": container with ID starting with c08466c61fcf40f08799375ccb6b8ee5d0398970285cf6407c7a961d840a5650 not found: ID does not exist"
Jul 15 10:59:29.686149 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.686126    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:29.690949 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:29.690926    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr"]
Jul 15 10:59:30.677226 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:30.675404    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" path="/var/lib/kubelet/pods/afe7ba8f-198a-45bd-b973-9c806e2c3d65/volumes"
Jul 15 10:59:32.828565 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:32.828526    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:38.348384 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.348352    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 10:59:38.348803 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.348788    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerName="istio-proxy"
Jul 15 10:59:38.348848 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.348804    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerName="istio-proxy"
Jul 15 10:59:38.348890 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.348873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe7ba8f-198a-45bd-b973-9c806e2c3d65" containerName="istio-proxy"
Jul 15 10:59:38.356688 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.356657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.359507 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.359486    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 10:59:38.360332 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.360317    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 10:59:38.360528 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.360511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 10:59:38.361251 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.361230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 10:59:38.361397 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.361278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 15 10:59:38.368141 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.368115    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 10:59:38.381075 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381221 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381221 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381221 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381413 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381413 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.381413 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.381289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.439192 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.439149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:38.482444 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482597 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482597 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482597 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482597 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482597 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482859 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.482957 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.483019 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.482989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.483098 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.483030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.483098 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.483079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.484926 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.484898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.485221 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.485203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.492409 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.492389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc\") pod \"precise-prefix-cache-test-kserve-7f794775fc-xwfcn\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.668108 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.667997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:38.821448 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.821422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 10:59:38.823410 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:59:38.823361    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b377ea7_9a91_4989_ae5e_d9427a27bab3.slice/crio-df668a729f4318f409d2b3b8c42e58194d51ca3573d89386e3ee1397097eee03 WatchSource:0}: Error finding container df668a729f4318f409d2b3b8c42e58194d51ca3573d89386e3ee1397097eee03: Status 404 returned error can't find the container with id df668a729f4318f409d2b3b8c42e58194d51ca3573d89386e3ee1397097eee03
Jul 15 10:59:38.936580 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.936505    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 10:59:38.941645 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.941626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.947364 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.947340    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-w5l7w\""
Jul 15 10:59:38.965655 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.965630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 10:59:38.985567 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.985691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.985691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6zs\" (UniqueName: \"kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.985691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.985691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:38.985691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:38.985684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.035732 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.035706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:39.086535 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086535 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6zs\" (UniqueName: \"kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086703 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.086960 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.086937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.087023 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.087005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.087088 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.087072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.087134 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.087112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.089230 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.089212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.099202 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.099176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6zs\" (UniqueName: \"kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.251146 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.251057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 10:59:39.406562 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.406539    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 10:59:39.408397 ip-10-0-137-92 kubenswrapper[2565]: W0715 10:59:39.408371    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de57973_f7a5_4ec9_88d7_22c1791f5d69.slice/crio-c4f5e0e9d549b06bad7e2b837cf011e8d1b2d72b721f7dc66662e5849d137c36 WatchSource:0}: Error finding container c4f5e0e9d549b06bad7e2b837cf011e8d1b2d72b721f7dc66662e5849d137c36: Status 404 returned error can't find the container with id c4f5e0e9d549b06bad7e2b837cf011e8d1b2d72b721f7dc66662e5849d137c36
Jul 15 10:59:39.716281 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.716238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerStarted","Data":"c4f5e0e9d549b06bad7e2b837cf011e8d1b2d72b721f7dc66662e5849d137c36"}
Jul 15 10:59:39.717691 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:39.717653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerStarted","Data":"df668a729f4318f409d2b3b8c42e58194d51ca3573d89386e3ee1397097eee03"}
Jul 15 10:59:41.776069 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:41.776010    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:42.521270 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:42.521234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:42.626884 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:42.626829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:43.747840 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:43.747732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerStarted","Data":"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"}
Jul 15 10:59:43.750074 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:43.750026    2565 generic.go:356] "Generic (PLEG): container finished" podID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerID="c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659" exitCode=0
Jul 15 10:59:43.750309 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:43.750076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659"}
Jul 15 10:59:46.772523 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:46.772485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerStarted","Data":"88f6446e92675a6ca705f3d1729bb6089fb9682adb7ba37a68b39082e8bdbf52"}
Jul 15 10:59:46.773931 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:46.773910    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerID="41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298" exitCode=0
Jul 15 10:59:46.774022 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:46.773950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerDied","Data":"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"}
Jul 15 10:59:46.999455 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:46.999375    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:49.800091 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:49.800028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerStarted","Data":"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"}
Jul 15 10:59:49.847346 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:49.847140    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" podStartSLOduration=2.402852831 podStartE2EDuration="11.847119706s" totalImagesPullingTime="6.467443081s" totalInitContainerRuntime="2.976823794s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:49.8467088 +0000 UTC m=+855.681453251" watchObservedRunningTime="2026-07-15 10:59:49.847119706 +0000 UTC m=+855.681864156"
Jul 15 10:59:50.676219 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:50.676188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:56.847030 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:56.847001    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/0.log"
Jul 15 10:59:56.847529 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:56.847492    2565 generic.go:356] "Generic (PLEG): container finished" podID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerID="88f6446e92675a6ca705f3d1729bb6089fb9682adb7ba37a68b39082e8bdbf52" exitCode=1
Jul 15 10:59:56.847606 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:56.847579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"88f6446e92675a6ca705f3d1729bb6089fb9682adb7ba37a68b39082e8bdbf52"}
Jul 15 10:59:58.311195 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:58.311162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 10:59:58.676327 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:58.676296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:58.684652 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:58.684625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 10:59:58.879535 ip-10-0-137-92 kubenswrapper[2565]: I0715 10:59:58.879502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 11:00:00.421180 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:00.421138    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:00:05.553301 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:05.553253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:00:05.890104 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:05.890069    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:00:15.882023 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:15.881988    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:00:23.021836 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:23.021810    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/0.log"
Jul 15 11:00:23.022284 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:23.022256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerStarted","Data":"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"}
Jul 15 11:00:23.022684 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:23.022663    2565 scope.go:117] "RemoveContainer" containerID="88f6446e92675a6ca705f3d1729bb6089fb9682adb7ba37a68b39082e8bdbf52"
Jul 15 11:00:24.030549 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:24.030520    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/0.log"
Jul 15 11:00:24.031069 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:24.031017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerStarted","Data":"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"}
Jul 15 11:00:24.031249 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:24.031229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:24.057719 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:24.057660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podStartSLOduration=3.787694171 podStartE2EDuration="46.05764457s" totalImagesPullingTime="41.641851207s" totalInitContainerRuntime="628.099192ms" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:00:24.054989598 +0000 UTC m=+889.889734077" watchObservedRunningTime="2026-07-15 11:00:24.05764457 +0000 UTC m=+889.892389020"
Jul 15 11:00:29.251431 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:29.251340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:29.251431 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:29.251411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:29.252468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:29.252438    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.45:8082/healthz\": dial tcp 10.134.0.45:8082: connect: connection refused"
Jul 15 11:00:29.252557 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:00:29.252470    2565 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:9003", ServerName: "10.134.0.45:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:9003: connect: connection refused"
Jul 15 11:00:30.251749 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:30.251701    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:9003\" within 1s: context deadline exceeded"
Jul 15 11:00:34.090740 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.090712    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:00:34.091494 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.091478    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/0.log"
Jul 15 11:00:34.091840 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.091818    2565 generic.go:356] "Generic (PLEG): container finished" podID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59" exitCode=1
Jul 15 11:00:34.091918 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.091899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"}
Jul 15 11:00:34.091976 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.091964    2565 scope.go:117] "RemoveContainer" containerID="88f6446e92675a6ca705f3d1729bb6089fb9682adb7ba37a68b39082e8bdbf52"
Jul 15 11:00:34.092389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.092365    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:00:34.092633 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:00:34.092615    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_kserve-ci-e2e-test(0de57973-f7a5-4ec9-88d7-22c1791f5d69)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69"
Jul 15 11:00:34.592928 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.592899    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:00:34.593253 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.593237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:00:34.611222 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.611198    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:00:34.611478 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.611462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:00:34.615205 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.615187    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:00:34.615426 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:34.615411    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:00:35.098684 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:35.098658    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:00:39.251376 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:39.251334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:39.251954 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:39.251935    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:00:39.252252 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:00:39.252228    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_kserve-ci-e2e-test(0de57973-f7a5-4ec9-88d7-22c1791f5d69)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69"
Jul 15 11:00:39.253218 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:39.253199    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:40.126252 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:40.126216    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:00:40.126581 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:00:40.126523    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_kserve-ci-e2e-test(0de57973-f7a5-4ec9-88d7-22c1791f5d69)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69"
Jul 15 11:00:40.127351 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:40.127327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:41.131460 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:41.131434    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:00:41.131850 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:00:41.131635    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_kserve-ci-e2e-test(0de57973-f7a5-4ec9-88d7-22c1791f5d69)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69"
Jul 15 11:00:49.869434 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:49.869399    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:00:52.673976 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:52.673951    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:00:53.195358 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:53.195326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:00:53.196081 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:53.196029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerStarted","Data":"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"}
Jul 15 11:00:53.196397 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:00:53.196374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:00:59.252499 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:00:59.252472    2565 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:9003", ServerName: "10.134.0.45:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:9003: connect: connection refused"
Jul 15 11:01:00.251951 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:00.251912    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:9003\" within 1s: context deadline exceeded"
Jul 15 11:01:24.203891 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:24.203857    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:01:26.327886 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:26.327853    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 11:01:26.342372 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:26.342340    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 11:01:26.342643 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:26.342597    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer" containerID="cri-o://6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1" gracePeriod=60
Jul 15 11:01:26.352779 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:26.352746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:28.391929 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:28.391900    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:01:28.392648 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:28.392626    2565 generic.go:356] "Generic (PLEG): container finished" podID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerID="6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1" exitCode=0
Jul 15 11:01:28.392738 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:28.392714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"}
Jul 15 11:01:30.096181 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:30.096144    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:30.126874 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:30.126839    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.45:8082/healthz\": dial tcp 10.134.0.45:8082: connect: connection refused"
Jul 15 11:01:33.058534 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:33.058497    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:33.191753 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:33.191717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:34.204512 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:34.204471    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:01:40.126802 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:40.126760    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.45:8082/healthz\": dial tcp 10.134.0.45:8082: connect: connection refused"
Jul 15 11:01:41.352640 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.352599    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="main" containerID="cri-o://2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9" gracePeriod=45
Jul 15 11:01:41.368507 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.368448    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main" containerID="cri-o://657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9" gracePeriod=45
Jul 15 11:01:41.636980 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.636950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 11:01:41.654177 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.654156    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:01:41.656521 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.656502    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:01:41.729870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.729826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730084 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.729912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730084 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.729937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730084 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.729955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730084 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730084 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730116    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache\") pod \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\" (UID: \"8b377ea7-9a91-4989-ae5e-d9427a27bab3\") "
Jul 15 11:01:41.730391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.730391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730247    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home" (OuterVolumeSpecName: "home") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.730562 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730424    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.730562 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730443    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.730669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.730646    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.732208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.732180    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm" (OuterVolumeSpecName: "dshm") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.732394 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.732246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc" (OuterVolumeSpecName: "kube-api-access-sq2sc") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "kube-api-access-sq2sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:01:41.732770 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.732750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:01:41.786834 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.786789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b377ea7-9a91-4989-ae5e-d9427a27bab3" (UID: "8b377ea7-9a91-4989-ae5e-d9427a27bab3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.831014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.830983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831125 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831028    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831180 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st6zs\" (UniqueName: \"kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831215 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs\") pod \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\" (UID: \"0de57973-f7a5-4ec9-88d7-22c1791f5d69\") "
Jul 15 11:01:41.831680 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.831680 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.831680 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.831486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.832121 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832100    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832126    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-tmp\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832142    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832155    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832170    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b377ea7-9a91-4989-ae5e-d9427a27bab3-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832183    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/8b377ea7-9a91-4989-ae5e-d9427a27bab3-kube-api-access-sq2sc\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832197    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832210    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b377ea7-9a91-4989-ae5e-d9427a27bab3-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.832508 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.832302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.833611 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.833587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:01:41.833611 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.833599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs" (OuterVolumeSpecName: "kube-api-access-st6zs") pod "0de57973-f7a5-4ec9-88d7-22c1791f5d69" (UID: "0de57973-f7a5-4ec9-88d7-22c1791f5d69"). InnerVolumeSpecName "kube-api-access-st6zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:01:41.933422 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.933333    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6zs\" (UniqueName: \"kubernetes.io/projected/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kube-api-access-st6zs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.933422 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.933367    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0de57973-f7a5-4ec9-88d7-22c1791f5d69-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.933422 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:41.933377    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de57973-f7a5-4ec9-88d7-22c1791f5d69-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:01:42.468276 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.468244    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerID="2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9" exitCode=0
Jul 15 11:01:42.468747 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.468317    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"
Jul 15 11:01:42.468747 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.468334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerDied","Data":"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"}
Jul 15 11:01:42.468747 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.468391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn" event={"ID":"8b377ea7-9a91-4989-ae5e-d9427a27bab3","Type":"ContainerDied","Data":"df668a729f4318f409d2b3b8c42e58194d51ca3573d89386e3ee1397097eee03"}
Jul 15 11:01:42.468747 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.468423    2565 scope.go:117] "RemoveContainer" containerID="2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"
Jul 15 11:01:42.470234 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.470218    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_0de57973-f7a5-4ec9-88d7-22c1791f5d69/main/1.log"
Jul 15 11:01:42.470934 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.470912    2565 generic.go:356] "Generic (PLEG): container finished" podID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerID="657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9" exitCode=0
Jul 15 11:01:42.471127 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.470950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"}
Jul 15 11:01:42.471127 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.470973    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"
Jul 15 11:01:42.471127 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.470993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq" event={"ID":"0de57973-f7a5-4ec9-88d7-22c1791f5d69","Type":"ContainerDied","Data":"c4f5e0e9d549b06bad7e2b837cf011e8d1b2d72b721f7dc66662e5849d137c36"}
Jul 15 11:01:42.478964 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.478948    2565 scope.go:117] "RemoveContainer" containerID="41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"
Jul 15 11:01:42.487443 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.487423    2565 scope.go:117] "RemoveContainer" containerID="2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"
Jul 15 11:01:42.487695 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.487677    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9\": container with ID starting with 2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9 not found: ID does not exist" containerID="2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"
Jul 15 11:01:42.487739 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.487704    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9"} err="failed to get container status \"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9\": rpc error: code = NotFound desc = could not find container \"2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9\": container with ID starting with 2458e8508638fabb56c329c86e0c11183aa5c0ad11d55535e4e0331d7ffa15c9 not found: ID does not exist"
Jul 15 11:01:42.487739 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.487720    2565 scope.go:117] "RemoveContainer" containerID="41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"
Jul 15 11:01:42.487946 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.487926    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298\": container with ID starting with 41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298 not found: ID does not exist" containerID="41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"
Jul 15 11:01:42.487988 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.487952    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298"} err="failed to get container status \"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298\": rpc error: code = NotFound desc = could not find container \"41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298\": container with ID starting with 41c37dfa83604d36ae6f71efaa3f3428b3584c70b6ad1ebbdcb7b908a0c6b298 not found: ID does not exist"
Jul 15 11:01:42.487988 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.487969    2565 scope.go:117] "RemoveContainer" containerID="657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"
Jul 15 11:01:42.496521 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.496502    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:01:42.497185 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.497037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 11:01:42.499498 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.499472    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn"]
Jul 15 11:01:42.504819 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.504804    2565 scope.go:117] "RemoveContainer" containerID="6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"
Jul 15 11:01:42.512797 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.512778    2565 scope.go:117] "RemoveContainer" containerID="c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659"
Jul 15 11:01:42.514295 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.514238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 11:01:42.520481 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.520462    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq"]
Jul 15 11:01:42.521284 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.521268    2565 scope.go:117] "RemoveContainer" containerID="657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"
Jul 15 11:01:42.521542 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.521522    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9\": container with ID starting with 657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9 not found: ID does not exist" containerID="657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"
Jul 15 11:01:42.521597 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.521548    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9"} err="failed to get container status \"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9\": rpc error: code = NotFound desc = could not find container \"657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9\": container with ID starting with 657b9846448adce5d99077daf2aaba815a8c5cbe3696e258b323d8459eab86a9 not found: ID does not exist"
Jul 15 11:01:42.521597 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.521567    2565 scope.go:117] "RemoveContainer" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:01:42.521808 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.521790    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59\": container with ID starting with 6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59 not found: ID does not exist" containerID="6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"
Jul 15 11:01:42.521851 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.521814    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59"} err="failed to get container status \"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59\": rpc error: code = NotFound desc = could not find container \"6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59\": container with ID starting with 6f412291866751d13d2834a1b88cf26657865551b25574f0096f517a3d082e59 not found: ID does not exist"
Jul 15 11:01:42.521851 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.521831    2565 scope.go:117] "RemoveContainer" containerID="6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"
Jul 15 11:01:42.522106 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.522028    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1\": container with ID starting with 6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1 not found: ID does not exist" containerID="6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"
Jul 15 11:01:42.522106 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.522089    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1"} err="failed to get container status \"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1\": rpc error: code = NotFound desc = could not find container \"6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1\": container with ID starting with 6927dfe4bb562ef8350fde5ec1ebfb3470b8de40734358e400864e98b761b5a1 not found: ID does not exist"
Jul 15 11:01:42.522106 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.522106    2565 scope.go:117] "RemoveContainer" containerID="c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659"
Jul 15 11:01:42.522334 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:01:42.522318    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659\": container with ID starting with c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659 not found: ID does not exist" containerID="c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659"
Jul 15 11:01:42.522373 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.522338    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659"} err="failed to get container status \"c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659\": rpc error: code = NotFound desc = could not find container \"c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659\": container with ID starting with c18d5bda22899bbba1944770397c8ebfda0c9bfe9475fd971bd3f36b8b7c6659 not found: ID does not exist"
Jul 15 11:01:42.674333 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.674250    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" path="/var/lib/kubelet/pods/0de57973-f7a5-4ec9-88d7-22c1791f5d69/volumes"
Jul 15 11:01:42.674736 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:42.674722    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" path="/var/lib/kubelet/pods/8b377ea7-9a91-4989-ae5e-d9427a27bab3/volumes"
Jul 15 11:01:51.359950 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.359913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:51.643183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643107    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:01:51.643478 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643465    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643478 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643478    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643491    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643497    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="main"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643510    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="main"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643520    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="storage-initializer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643525    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="storage-initializer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643531    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="storage-initializer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643536    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="storage-initializer"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643541    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643546    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643599    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b377ea7-9a91-4989-ae5e-d9427a27bab3" containerName="main"
Jul 15 11:01:51.643885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643605    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643611    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:01:51.643885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.643615    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="tokenizer"
Jul 15 11:01:51.647458 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.647428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.650648 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.650623    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:01:51.651868 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.651830    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:01:51.652514 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.652125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:01:51.655122 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.652768    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 15 11:01:51.655122 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.653159    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-74wk5\""
Jul 15 11:01:51.663476 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.663452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:01:51.693926 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.693898    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:01:51.716941 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.716916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.717058 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.716990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.717058 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.717021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2kx\" (UniqueName: \"kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.818319 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.818268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.818503 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.818359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.818503 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.818386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2kx\" (UniqueName: \"kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.818835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.818805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.820917 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.820894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.827186 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.827161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2kx\" (UniqueName: \"kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:51.964242 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:51.964145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:52.101858 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:52.101828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:01:52.103306 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:01:52.103283    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cdc6d2_d454_4629_8e4c_7761c5b688c3.slice/crio-0c74a46a23b9aeb661dbcf16e20d5f313acad046d28843122be96506eed3d4f5 WatchSource:0}: Error finding container 0c74a46a23b9aeb661dbcf16e20d5f313acad046d28843122be96506eed3d4f5: Status 404 returned error can't find the container with id 0c74a46a23b9aeb661dbcf16e20d5f313acad046d28843122be96506eed3d4f5
Jul 15 11:01:52.529065 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:52.529008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" event={"ID":"90cdc6d2-d454-4629-8e4c-7761c5b688c3","Type":"ContainerStarted","Data":"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"}
Jul 15 11:01:52.529495 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:52.529095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" event={"ID":"90cdc6d2-d454-4629-8e4c-7761c5b688c3","Type":"ContainerStarted","Data":"0c74a46a23b9aeb661dbcf16e20d5f313acad046d28843122be96506eed3d4f5"}
Jul 15 11:01:52.529495 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:52.529202    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:01:52.547715 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:01:52.547661    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" podStartSLOduration=1.547640931 podStartE2EDuration="1.547640931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:01:52.547009834 +0000 UTC m=+978.381754300" watchObservedRunningTime="2026-07-15 11:01:52.547640931 +0000 UTC m=+978.382385377"
Jul 15 11:02:09.244204 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:09.244167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:23.537641 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:23.537614    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:02:24.785622 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:24.785586    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:02:24.793378 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:24.793335    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:24.863668 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:24.863633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:33.537267 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:33.537222    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:02:39.806771 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:39.806733    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerName="main" containerID="cri-o://fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4" gracePeriod=45
Jul 15 11:02:40.074989 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.074963    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:02:40.204564 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.204528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds\") pod \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") "
Jul 15 11:02:40.204764 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.204608    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr2kx\" (UniqueName: \"kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx\") pod \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") "
Jul 15 11:02:40.204764 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.204688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs\") pod \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\" (UID: \"90cdc6d2-d454-4629-8e4c-7761c5b688c3\") "
Jul 15 11:02:40.204894 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.204839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "90cdc6d2-d454-4629-8e4c-7761c5b688c3" (UID: "90cdc6d2-d454-4629-8e4c-7761c5b688c3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:40.204971 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.204948    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:02:40.206860 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.206822    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx" (OuterVolumeSpecName: "kube-api-access-rr2kx") pod "90cdc6d2-d454-4629-8e4c-7761c5b688c3" (UID: "90cdc6d2-d454-4629-8e4c-7761c5b688c3"). InnerVolumeSpecName "kube-api-access-rr2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:02:40.206963 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.206920    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90cdc6d2-d454-4629-8e4c-7761c5b688c3" (UID: "90cdc6d2-d454-4629-8e4c-7761c5b688c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:02:40.305663 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.305621    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdc6d2-d454-4629-8e4c-7761c5b688c3-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:02:40.305663 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.305657    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2kx\" (UniqueName: \"kubernetes.io/projected/90cdc6d2-d454-4629-8e4c-7761c5b688c3-kube-api-access-rr2kx\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:02:40.526724 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.526690    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:40.600222 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.600192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:40.804333 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.804242    2565 generic.go:356] "Generic (PLEG): container finished" podID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerID="fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4" exitCode=0
Jul 15 11:02:40.804333 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.804300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"
Jul 15 11:02:40.804513 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.804331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" event={"ID":"90cdc6d2-d454-4629-8e4c-7761c5b688c3","Type":"ContainerDied","Data":"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"}
Jul 15 11:02:40.804513 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.804388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2" event={"ID":"90cdc6d2-d454-4629-8e4c-7761c5b688c3","Type":"ContainerDied","Data":"0c74a46a23b9aeb661dbcf16e20d5f313acad046d28843122be96506eed3d4f5"}
Jul 15 11:02:40.804513 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.804416    2565 scope.go:117] "RemoveContainer" containerID="fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"
Jul 15 11:02:40.813745 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.813512    2565 scope.go:117] "RemoveContainer" containerID="fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"
Jul 15 11:02:40.813972 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:02:40.813802    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4\": container with ID starting with fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4 not found: ID does not exist" containerID="fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"
Jul 15 11:02:40.813972 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.813835    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4"} err="failed to get container status \"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4\": rpc error: code = NotFound desc = could not find container \"fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4\": container with ID starting with fcb4c93347d74b0444d0857118ed4eb052441e4187db3b4a3f7903785dc33ec4 not found: ID does not exist"
Jul 15 11:02:40.824625 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.824599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:02:40.830376 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:40.830354    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2"]
Jul 15 11:02:42.674669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:42.674628    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" path="/var/lib/kubelet/pods/90cdc6d2-d454-4629-8e4c-7761c5b688c3/volumes"
Jul 15 11:02:47.953619 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:47.953587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:48.240192 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240114    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:02:48.240557 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240543    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:02:48.240599 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:02:48.240599 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240585    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerName="main"
Jul 15 11:02:48.240599 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240590    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerName="main"
Jul 15 11:02:48.240703 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240650    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de57973-f7a5-4ec9-88d7-22c1791f5d69" containerName="main"
Jul 15 11:02:48.240703 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.240661    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cdc6d2-d454-4629-8e4c-7761c5b688c3" containerName="main"
Jul 15 11:02:48.243536 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.243519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.247282 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.247257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:02:48.247427 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.247258    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:02:48.247427 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.247258    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-gtvn9\""
Jul 15 11:02:48.247549 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.247287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 15 11:02:48.247549 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.247309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:02:48.254733 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.254714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:02:48.276818 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.276784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:02:48.374104 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.374026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.374292 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.374113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.374292 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.374259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t49\" (UniqueName: \"kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.475845 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.475812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.475845 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.475849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.476077 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.475904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t49\" (UniqueName: \"kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.476323 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.476301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.478555 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.478534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.486764 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.486736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t49\" (UniqueName: \"kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.553598 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.553512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.692989 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.692957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:02:48.694414 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:02:48.694390    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504945f2_ea6d_4ed2_9d96_53181b4eda37.slice/crio-3db05bc7dd4e8d428ee8f285c414eeb2581781568e375fda6c238a8a712c556b WatchSource:0}: Error finding container 3db05bc7dd4e8d428ee8f285c414eeb2581781568e375fda6c238a8a712c556b: Status 404 returned error can't find the container with id 3db05bc7dd4e8d428ee8f285c414eeb2581781568e375fda6c238a8a712c556b
Jul 15 11:02:48.853533 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.853497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" event={"ID":"504945f2-ea6d-4ed2-9d96-53181b4eda37","Type":"ContainerStarted","Data":"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"}
Jul 15 11:02:48.853679 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.853551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" event={"ID":"504945f2-ea6d-4ed2-9d96-53181b4eda37","Type":"ContainerStarted","Data":"3db05bc7dd4e8d428ee8f285c414eeb2581781568e375fda6c238a8a712c556b"}
Jul 15 11:02:48.853679 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.853578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:02:48.872101 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:02:48.872051    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" podStartSLOduration=0.872024044 podStartE2EDuration="872.024044ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:02:48.869736608 +0000 UTC m=+1034.704481118" watchObservedRunningTime="2026-07-15 11:02:48.872024044 +0000 UTC m=+1034.706768493"
Jul 15 11:03:13.418685 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:13.418651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:19.860983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:19.860949    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:03:22.638452 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:22.638404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:22.858453 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:22.858422    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:37.415531 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:37.415501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:57.931622 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:57.931586    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:58.229300 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.229206    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:03:58.234033 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.234009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.238686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.238660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-8w7wz\""
Jul 15 11:03:58.239008 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.238981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 15 11:03:58.242424 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.242402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:03:58.301166 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.301131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.301369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.301182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sf74\" (UniqueName: \"kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.301369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.301219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.402332 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.402287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.402530 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.402430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.402530 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.402462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf74\" (UniqueName: \"kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.402831 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.402806    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.404948 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.404930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.411968 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.411931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf74\" (UniqueName: \"kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.544650 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.544555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:58.691329 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.691305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:03:58.693514 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:03:58.693491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c6beec_fb45_4a15_8656_bdd49f9f1905.slice/crio-646a891dd537eb4694df5acc3308bd2854efb38de7b533e4ecd79304b9f72280 WatchSource:0}: Error finding container 646a891dd537eb4694df5acc3308bd2854efb38de7b533e4ecd79304b9f72280: Status 404 returned error can't find the container with id 646a891dd537eb4694df5acc3308bd2854efb38de7b533e4ecd79304b9f72280
Jul 15 11:03:58.725450 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:58.725421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:03:59.227799 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:59.227762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" event={"ID":"c2c6beec-fb45-4a15-8656-bdd49f9f1905","Type":"ContainerStarted","Data":"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"}
Jul 15 11:03:59.228210 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:59.227819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" event={"ID":"c2c6beec-fb45-4a15-8656-bdd49f9f1905","Type":"ContainerStarted","Data":"646a891dd537eb4694df5acc3308bd2854efb38de7b533e4ecd79304b9f72280"}
Jul 15 11:03:59.228210 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:59.227846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:03:59.250961 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:03:59.250901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" podStartSLOduration=1.250885603 podStartE2EDuration="1.250885603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:03:59.246853409 +0000 UTC m=+1105.081597884" watchObservedRunningTime="2026-07-15 11:03:59.250885603 +0000 UTC m=+1105.085630052"
Jul 15 11:04:18.452078 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:18.452017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:24.032722 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:24.032680    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:30.234700 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:30.234668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:04:31.391121 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:31.391083    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:31.392186 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:31.392163    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:04:31.472628 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:31.472591    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:40.234449 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:40.234415    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:04:46.413895 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.413829    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerName="main" containerID="cri-o://0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1" gracePeriod=45
Jul 15 11:04:46.673607 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.673536    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:04:46.721411 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.721383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds\") pod \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") "
Jul 15 11:04:46.721566 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.721429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sf74\" (UniqueName: \"kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74\") pod \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") "
Jul 15 11:04:46.721566 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.721507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs\") pod \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\" (UID: \"c2c6beec-fb45-4a15-8656-bdd49f9f1905\") "
Jul 15 11:04:46.721638 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.721608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c2c6beec-fb45-4a15-8656-bdd49f9f1905" (UID: "c2c6beec-fb45-4a15-8656-bdd49f9f1905"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:04:46.721764 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.721747    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:04:46.723609 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.723582    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2c6beec-fb45-4a15-8656-bdd49f9f1905" (UID: "c2c6beec-fb45-4a15-8656-bdd49f9f1905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:04:46.723714 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.723662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74" (OuterVolumeSpecName: "kube-api-access-8sf74") pod "c2c6beec-fb45-4a15-8656-bdd49f9f1905" (UID: "c2c6beec-fb45-4a15-8656-bdd49f9f1905"). InnerVolumeSpecName "kube-api-access-8sf74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:04:46.822501 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.822462    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c6beec-fb45-4a15-8656-bdd49f9f1905-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:04:46.822501 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:46.822495    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sf74\" (UniqueName: \"kubernetes.io/projected/c2c6beec-fb45-4a15-8656-bdd49f9f1905-kube-api-access-8sf74\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:04:47.483703 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.483665    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerID="0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1" exitCode=0
Jul 15 11:04:47.484140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.483711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"
Jul 15 11:04:47.484140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.483711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" event={"ID":"c2c6beec-fb45-4a15-8656-bdd49f9f1905","Type":"ContainerDied","Data":"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"}
Jul 15 11:04:47.484140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.483766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk" event={"ID":"c2c6beec-fb45-4a15-8656-bdd49f9f1905","Type":"ContainerDied","Data":"646a891dd537eb4694df5acc3308bd2854efb38de7b533e4ecd79304b9f72280"}
Jul 15 11:04:47.484140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.483794    2565 scope.go:117] "RemoveContainer" containerID="0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"
Jul 15 11:04:47.493485 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.493464    2565 scope.go:117] "RemoveContainer" containerID="0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"
Jul 15 11:04:47.493747 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:04:47.493726    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1\": container with ID starting with 0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1 not found: ID does not exist" containerID="0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"
Jul 15 11:04:47.493821 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.493756    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1"} err="failed to get container status \"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1\": rpc error: code = NotFound desc = could not find container \"0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1\": container with ID starting with 0d5b9124421778a3782cd20e2b76ee079559fd66bf690e42d12e8e86ceafe1b1 not found: ID does not exist"
Jul 15 11:04:47.506784 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.506752    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:04:47.510777 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:47.510751    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk"]
Jul 15 11:04:48.674395 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:48.674360    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" path="/var/lib/kubelet/pods/c2c6beec-fb45-4a15-8656-bdd49f9f1905/volumes"
Jul 15 11:04:51.862985 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:51.862941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:51.868116 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:51.868087    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:04:51.926658 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:51.926627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:04:59.860778 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:04:59.860740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:05:06.889479 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:06.889415    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerName="main" containerID="cri-o://8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b" gracePeriod=45
Jul 15 11:05:07.139182 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.139154    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:05:07.210274 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.210197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs\") pod \"504945f2-ea6d-4ed2-9d96-53181b4eda37\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") "
Jul 15 11:05:07.210274 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.210238    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds\") pod \"504945f2-ea6d-4ed2-9d96-53181b4eda37\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") "
Jul 15 11:05:07.210452 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.210288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4t49\" (UniqueName: \"kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49\") pod \"504945f2-ea6d-4ed2-9d96-53181b4eda37\" (UID: \"504945f2-ea6d-4ed2-9d96-53181b4eda37\") "
Jul 15 11:05:07.210522 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.210500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "504945f2-ea6d-4ed2-9d96-53181b4eda37" (UID: "504945f2-ea6d-4ed2-9d96-53181b4eda37"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:07.210587 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.210568    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/504945f2-ea6d-4ed2-9d96-53181b4eda37-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:05:07.212493 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.212460    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "504945f2-ea6d-4ed2-9d96-53181b4eda37" (UID: "504945f2-ea6d-4ed2-9d96-53181b4eda37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:05:07.212606 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.212536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49" (OuterVolumeSpecName: "kube-api-access-s4t49") pod "504945f2-ea6d-4ed2-9d96-53181b4eda37" (UID: "504945f2-ea6d-4ed2-9d96-53181b4eda37"). InnerVolumeSpecName "kube-api-access-s4t49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:05:07.311579 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.311536    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4t49\" (UniqueName: \"kubernetes.io/projected/504945f2-ea6d-4ed2-9d96-53181b4eda37-kube-api-access-s4t49\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:05:07.311579 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.311570    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504945f2-ea6d-4ed2-9d96-53181b4eda37-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:05:07.591197 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.591164    2565 generic.go:356] "Generic (PLEG): container finished" podID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerID="8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b" exitCode=0
Jul 15 11:05:07.591362 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.591219    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"
Jul 15 11:05:07.591362 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.591242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" event={"ID":"504945f2-ea6d-4ed2-9d96-53181b4eda37","Type":"ContainerDied","Data":"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"}
Jul 15 11:05:07.591362 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.591284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb" event={"ID":"504945f2-ea6d-4ed2-9d96-53181b4eda37","Type":"ContainerDied","Data":"3db05bc7dd4e8d428ee8f285c414eeb2581781568e375fda6c238a8a712c556b"}
Jul 15 11:05:07.591362 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.591303    2565 scope.go:117] "RemoveContainer" containerID="8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"
Jul 15 11:05:07.600554 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.600535    2565 scope.go:117] "RemoveContainer" containerID="8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"
Jul 15 11:05:07.600834 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:05:07.600814    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b\": container with ID starting with 8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b not found: ID does not exist" containerID="8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"
Jul 15 11:05:07.600904 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.600845    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b"} err="failed to get container status \"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b\": rpc error: code = NotFound desc = could not find container \"8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b\": container with ID starting with 8c3e97f0cf292d486a10e09301bdb94f4b825cd292012a27303dc9f9bd5a986b not found: ID does not exist"
Jul 15 11:05:07.613524 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.613491    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:05:07.618369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:07.618349    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb"]
Jul 15 11:05:08.675608 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:08.675569    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" path="/var/lib/kubelet/pods/504945f2-ea6d-4ed2-9d96-53181b4eda37/volumes"
Jul 15 11:05:15.658357 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.658318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:15.928602 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.928514    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:05:15.929062 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929017    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerName="main"
Jul 15 11:05:15.929062 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerName="main"
Jul 15 11:05:15.929220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929078    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerName="main"
Jul 15 11:05:15.929220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerName="main"
Jul 15 11:05:15.929220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2c6beec-fb45-4a15-8656-bdd49f9f1905" containerName="main"
Jul 15 11:05:15.929220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.929165    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="504945f2-ea6d-4ed2-9d96-53181b4eda37" containerName="main"
Jul 15 11:05:15.936576 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.934165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:15.938159 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.938132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:05:15.938159 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.938148    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:05:15.938371 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.938160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 15 11:05:15.938371 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.938266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-zp6db\""
Jul 15 11:05:15.938371 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.938279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:05:15.945870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.945850    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:05:15.964103 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:15.964065    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:16.097805 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.097760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.097991 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.097874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.097991 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.097930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdd2\" (UniqueName: \"kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.198673 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.198570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.198673 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.198633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdd2\" (UniqueName: \"kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.198673 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.198664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.199145 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.199111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.201589 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.201567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.213794 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.213770    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdd2\" (UniqueName: \"kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.246231 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.246190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.389861 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.389835    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:05:16.391799 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:05:16.391770    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd630aff8_8bff_4117_9ecc_dc4e984b015d.slice/crio-78ca1a1fef1caf850df545a7a7e184ae3f486baebf2f8669c19bb83c0bbf169f WatchSource:0}: Error finding container 78ca1a1fef1caf850df545a7a7e184ae3f486baebf2f8669c19bb83c0bbf169f: Status 404 returned error can't find the container with id 78ca1a1fef1caf850df545a7a7e184ae3f486baebf2f8669c19bb83c0bbf169f
Jul 15 11:05:16.642062 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.642020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" event={"ID":"d630aff8-8bff-4117-9ecc-dc4e984b015d","Type":"ContainerStarted","Data":"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"}
Jul 15 11:05:16.642259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.642098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" event={"ID":"d630aff8-8bff-4117-9ecc-dc4e984b015d","Type":"ContainerStarted","Data":"78ca1a1fef1caf850df545a7a7e184ae3f486baebf2f8669c19bb83c0bbf169f"}
Jul 15 11:05:16.642259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.642166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:16.664647 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:16.664413    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" podStartSLOduration=1.664393174 podStartE2EDuration="1.664393174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:05:16.662896152 +0000 UTC m=+1182.497640627" watchObservedRunningTime="2026-07-15 11:05:16.664393174 +0000 UTC m=+1182.499137625"
Jul 15 11:05:34.652274 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:34.652248    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:05:34.652836 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:34.652331    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:05:34.656725 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:34.656705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:05:34.656725 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:34.656714    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:05:36.124813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:36.124779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:39.294298 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:39.294251    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:39.388906 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:39.388865    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:47.649819 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:47.649791    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:05:53.722863 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:53.722823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:53.870367 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:53.870331    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:58.520208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:58.520170    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:05:58.622869 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:05:58.622835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:10.828276 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:10.828240    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:20.294619 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.294589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:20.571430 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.571393    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:06:20.574735 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.574715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.577547 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.577523    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-gstd4\""
Jul 15 11:06:20.577680 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.577575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:06:20.577742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.577676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 15 11:06:20.589279 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.589258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:06:20.615488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.615459    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:20.663283 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.663249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5gm\" (UniqueName: \"kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.663455 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.663307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.663455 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.663381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.663455 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.663428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.764455 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.764414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5gm\" (UniqueName: \"kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.764655 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.764471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.764725 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.764645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.764725 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.764693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.764852 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.764832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.765018 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.765001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.767424 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.767399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.775297 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.775270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5gm\" (UniqueName: \"kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm\") pod \"auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:20.884747 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:20.884652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:21.031194 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:21.031165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:06:21.032141 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:06:21.032112    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda327ce82_c847_4990_920d_31185e4adf47.slice/crio-1574c918465475eb77b8b1be3c9d4679bab65118aa8bc7799ca7454a68e9bcb5 WatchSource:0}: Error finding container 1574c918465475eb77b8b1be3c9d4679bab65118aa8bc7799ca7454a68e9bcb5: Status 404 returned error can't find the container with id 1574c918465475eb77b8b1be3c9d4679bab65118aa8bc7799ca7454a68e9bcb5
Jul 15 11:06:21.996779 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:21.996686    2565 generic.go:356] "Generic (PLEG): container finished" podID="a327ce82-c847-4990-920d-31185e4adf47" containerID="7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006" exitCode=0
Jul 15 11:06:21.996779 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:21.996729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" event={"ID":"a327ce82-c847-4990-920d-31185e4adf47","Type":"ContainerDied","Data":"7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006"}
Jul 15 11:06:21.997174 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:21.996799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" event={"ID":"a327ce82-c847-4990-920d-31185e4adf47","Type":"ContainerStarted","Data":"1574c918465475eb77b8b1be3c9d4679bab65118aa8bc7799ca7454a68e9bcb5"}
Jul 15 11:06:23.004028 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:23.003992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" event={"ID":"a327ce82-c847-4990-920d-31185e4adf47","Type":"ContainerStarted","Data":"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"}
Jul 15 11:06:23.004409 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:23.004106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:23.025480 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:23.025373    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" podStartSLOduration=2.369081882 podStartE2EDuration="3.02535434s" totalImagesPullingTime="0s" totalInitContainerRuntime="656.272458ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:06:23.024961343 +0000 UTC m=+1248.859705819" watchObservedRunningTime="2026-07-15 11:06:23.02535434 +0000 UTC m=+1248.860098791"
Jul 15 11:06:33.758766 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:33.758683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:35.901799 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:35.901764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:37.794068 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:37.790481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:38.544693 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:38.544651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:54.011087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:54.011024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:06:57.977678 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:57.977641    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:06:57.995889 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:57.995821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:06:58.061665 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:06:58.061627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:07:07.649783 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:07.649740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:07:13.000789 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.000729    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerName="main" containerID="cri-o://d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51" gracePeriod=45
Jul 15 11:07:13.247213 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.247185    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:07:13.273804 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.273719    2565 generic.go:356] "Generic (PLEG): container finished" podID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerID="d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51" exitCode=0
Jul 15 11:07:13.273804 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.273766    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"
Jul 15 11:07:13.273983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.273765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" event={"ID":"d630aff8-8bff-4117-9ecc-dc4e984b015d","Type":"ContainerDied","Data":"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"}
Jul 15 11:07:13.273983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.273922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd" event={"ID":"d630aff8-8bff-4117-9ecc-dc4e984b015d","Type":"ContainerDied","Data":"78ca1a1fef1caf850df545a7a7e184ae3f486baebf2f8669c19bb83c0bbf169f"}
Jul 15 11:07:13.273983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.273965    2565 scope.go:117] "RemoveContainer" containerID="d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"
Jul 15 11:07:13.284583 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.284564    2565 scope.go:117] "RemoveContainer" containerID="d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"
Jul 15 11:07:13.284878 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:07:13.284861    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51\": container with ID starting with d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51 not found: ID does not exist" containerID="d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"
Jul 15 11:07:13.284956 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.284885    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51"} err="failed to get container status \"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51\": rpc error: code = NotFound desc = could not find container \"d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51\": container with ID starting with d4d6a8499d69c9991a8539eda868959d8f049e6f370aac83a078aef9d5a31a51 not found: ID does not exist"
Jul 15 11:07:13.306277 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.306254    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttdd2\" (UniqueName: \"kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2\") pod \"d630aff8-8bff-4117-9ecc-dc4e984b015d\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") "
Jul 15 11:07:13.306395 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.306318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs\") pod \"d630aff8-8bff-4117-9ecc-dc4e984b015d\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") "
Jul 15 11:07:13.306395 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.306347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds\") pod \"d630aff8-8bff-4117-9ecc-dc4e984b015d\" (UID: \"d630aff8-8bff-4117-9ecc-dc4e984b015d\") "
Jul 15 11:07:13.306681 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.306659    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d630aff8-8bff-4117-9ecc-dc4e984b015d" (UID: "d630aff8-8bff-4117-9ecc-dc4e984b015d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:07:13.308515 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.308494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d630aff8-8bff-4117-9ecc-dc4e984b015d" (UID: "d630aff8-8bff-4117-9ecc-dc4e984b015d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:07:13.308578 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.308512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2" (OuterVolumeSpecName: "kube-api-access-ttdd2") pod "d630aff8-8bff-4117-9ecc-dc4e984b015d" (UID: "d630aff8-8bff-4117-9ecc-dc4e984b015d"). InnerVolumeSpecName "kube-api-access-ttdd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:07:13.407920 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.407884    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d630aff8-8bff-4117-9ecc-dc4e984b015d-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:07:13.407920 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.407915    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d630aff8-8bff-4117-9ecc-dc4e984b015d-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:07:13.407920 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.407926    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdd2\" (UniqueName: \"kubernetes.io/projected/d630aff8-8bff-4117-9ecc-dc4e984b015d-kube-api-access-ttdd2\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:07:13.597774 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.597744    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:07:13.603903 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:13.603880    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd"]
Jul 15 11:07:14.675511 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:14.675476    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" path="/var/lib/kubelet/pods/d630aff8-8bff-4117-9ecc-dc4e984b015d/volumes"
Jul 15 11:07:25.306508 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.306475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:07:25.665994 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.665958    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:07:25.666424 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.666410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerName="main"
Jul 15 11:07:25.666472 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.666427    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerName="main"
Jul 15 11:07:25.666507 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.666501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d630aff8-8bff-4117-9ecc-dc4e984b015d" containerName="main"
Jul 15 11:07:25.670905 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.670878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.674089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.674065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 15 11:07:25.674227 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.674108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-xprvv\""
Jul 15 11:07:25.683822 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.683795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:07:25.696983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.696946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:07:25.713249 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.713201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.713439 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.713259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.713439 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.713285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.713439 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.713305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjld\" (UniqueName: \"kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.813865 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.813830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.814067 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.813873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.814067 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.813898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.814067 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.813920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjld\" (UniqueName: \"kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.814344 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.814319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.814386 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.814331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.816481 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.816462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.822714 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.822688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjld\" (UniqueName: \"kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:25.980669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:25.980583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:26.122821 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:26.122792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:07:26.124085 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:07:26.124056    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edb146b_c10e_461d_9c77_ddf0bfa148d7.slice/crio-5ea53083cd16ca5a253f1ec24b93922b2398f6df7e880b2932bea57aa6d661af WatchSource:0}: Error finding container 5ea53083cd16ca5a253f1ec24b93922b2398f6df7e880b2932bea57aa6d661af: Status 404 returned error can't find the container with id 5ea53083cd16ca5a253f1ec24b93922b2398f6df7e880b2932bea57aa6d661af
Jul 15 11:07:26.345366 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:26.345328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerStarted","Data":"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"}
Jul 15 11:07:26.345851 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:26.345400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerStarted","Data":"5ea53083cd16ca5a253f1ec24b93922b2398f6df7e880b2932bea57aa6d661af"}
Jul 15 11:07:27.353443 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:27.353411    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerID="b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb" exitCode=0
Jul 15 11:07:27.353832 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:27.353482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerDied","Data":"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"}
Jul 15 11:07:28.360289 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:28.360251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerStarted","Data":"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"}
Jul 15 11:07:28.360704 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:28.360351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:07:28.382305 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:28.382237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" podStartSLOduration=2.729542039 podStartE2EDuration="3.382214279s" totalImagesPullingTime="0s" totalInitContainerRuntime="652.67224ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:28.381415879 +0000 UTC m=+1314.216160329" watchObservedRunningTime="2026-07-15 11:07:28.382214279 +0000 UTC m=+1314.216958742"
Jul 15 11:07:36.019756 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:36.019713    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:07:59.367871 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:07:59.367796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:08:25.141694 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:25.141662    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:08:25.145722 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:25.145689    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:08:25.212369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:25.212336    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:08:34.010995 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:34.010949    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:08:40.165906 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.165851    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="main" containerID="cri-o://fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f" gracePeriod=45
Jul 15 11:08:40.440730 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.440701    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:08:40.576961 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.576922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz5gm\" (UniqueName: \"kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm\") pod \"a327ce82-c847-4990-920d-31185e4adf47\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") "
Jul 15 11:08:40.576961 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.576962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds\") pod \"a327ce82-c847-4990-920d-31185e4adf47\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") "
Jul 15 11:08:40.577220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.577027    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs\") pod \"a327ce82-c847-4990-920d-31185e4adf47\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") "
Jul 15 11:08:40.577220 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.577087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location\") pod \"a327ce82-c847-4990-920d-31185e4adf47\" (UID: \"a327ce82-c847-4990-920d-31185e4adf47\") "
Jul 15 11:08:40.577287 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.577260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a327ce82-c847-4990-920d-31185e4adf47" (UID: "a327ce82-c847-4990-920d-31185e4adf47"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:40.577410 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.577389    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:40.577830 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.577760    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a327ce82-c847-4990-920d-31185e4adf47" (UID: "a327ce82-c847-4990-920d-31185e4adf47"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:40.579294 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.579267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a327ce82-c847-4990-920d-31185e4adf47" (UID: "a327ce82-c847-4990-920d-31185e4adf47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:08:40.579393 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.579351    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm" (OuterVolumeSpecName: "kube-api-access-mz5gm") pod "a327ce82-c847-4990-920d-31185e4adf47" (UID: "a327ce82-c847-4990-920d-31185e4adf47"). InnerVolumeSpecName "kube-api-access-mz5gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:08:40.678925 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.678839    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5gm\" (UniqueName: \"kubernetes.io/projected/a327ce82-c847-4990-920d-31185e4adf47-kube-api-access-mz5gm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:40.678925 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.678876    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a327ce82-c847-4990-920d-31185e4adf47-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:40.678925 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.678895    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a327ce82-c847-4990-920d-31185e4adf47-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:40.773893 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.773851    2565 generic.go:356] "Generic (PLEG): container finished" podID="a327ce82-c847-4990-920d-31185e4adf47" containerID="fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f" exitCode=0
Jul 15 11:08:40.774108 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.773909    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"
Jul 15 11:08:40.774108 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.773936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" event={"ID":"a327ce82-c847-4990-920d-31185e4adf47","Type":"ContainerDied","Data":"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"}
Jul 15 11:08:40.774108 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.773993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt" event={"ID":"a327ce82-c847-4990-920d-31185e4adf47","Type":"ContainerDied","Data":"1574c918465475eb77b8b1be3c9d4679bab65118aa8bc7799ca7454a68e9bcb5"}
Jul 15 11:08:40.774108 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.774018    2565 scope.go:117] "RemoveContainer" containerID="fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"
Jul 15 11:08:40.783247 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.783225    2565 scope.go:117] "RemoveContainer" containerID="7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006"
Jul 15 11:08:40.792960 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.792925    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:08:40.793247 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.793203    2565 scope.go:117] "RemoveContainer" containerID="fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"
Jul 15 11:08:40.793656 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:08:40.793616    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f\": container with ID starting with fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f not found: ID does not exist" containerID="fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"
Jul 15 11:08:40.793829 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.793793    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f"} err="failed to get container status \"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f\": rpc error: code = NotFound desc = could not find container \"fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f\": container with ID starting with fbbfb920f8c184e52fd70c56ffc20dc2c7f40c203164d35304e48ca70146733f not found: ID does not exist"
Jul 15 11:08:40.793923 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.793832    2565 scope.go:117] "RemoveContainer" containerID="7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006"
Jul 15 11:08:40.794400 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:08:40.794374    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006\": container with ID starting with 7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006 not found: ID does not exist" containerID="7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006"
Jul 15 11:08:40.794478 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.794408    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006"} err="failed to get container status \"7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006\": rpc error: code = NotFound desc = could not find container \"7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006\": container with ID starting with 7cbf6aae52660bf1c71b9dff41e3e0440560b9b858368399331149c60250d006 not found: ID does not exist"
Jul 15 11:08:40.796276 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:40.796259    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt"]
Jul 15 11:08:42.674545 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:42.674509    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a327ce82-c847-4990-920d-31185e4adf47" path="/var/lib/kubelet/pods/a327ce82-c847-4990-920d-31185e4adf47/volumes"
Jul 15 11:08:45.904205 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904164    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:45.904686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904560    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="main"
Jul 15 11:08:45.904686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="main"
Jul 15 11:08:45.904686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904588    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="storage-initializer"
Jul 15 11:08:45.904686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904594    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="storage-initializer"
Jul 15 11:08:45.904686 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.904652    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a327ce82-c847-4990-920d-31185e4adf47" containerName="main"
Jul 15 11:08:45.909484 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.909461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:45.911958 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.911936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-95m52\""
Jul 15 11:08:45.912127 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.911936    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 11:08:45.919300 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:45.919277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:46.025495 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.025848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.025780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2c9\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127204 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127204 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2c9\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127820 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127820 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127912 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.127947 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.127925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.128136 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.128117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.130430 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.130402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.130649 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.130631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.142213 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.142180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.144897 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.144873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2c9\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9\") pod \"router-gateway-1-openshift-default-6679d9568-m7687\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.220901 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.220812    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:46.369349 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.369322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:46.370960 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:08:46.370929    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda3d049_a122_4123_98c2_cbfa5b51f4ba.slice/crio-2aa32da1909d5274bf6f365810becfc6aaa00e461d5da1d293d26a46fc9fcfef WatchSource:0}: Error finding container 2aa32da1909d5274bf6f365810becfc6aaa00e461d5da1d293d26a46fc9fcfef: Status 404 returned error can't find the container with id 2aa32da1909d5274bf6f365810becfc6aaa00e461d5da1d293d26a46fc9fcfef
Jul 15 11:08:46.373078 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.373010    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:08:46.373157 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.373103    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:08:46.373157 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.373136    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:08:46.589720 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.589685    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:46.810010 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.809976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" event={"ID":"bda3d049-a122-4123-98c2-cbfa5b51f4ba","Type":"ContainerStarted","Data":"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"}
Jul 15 11:08:46.810198 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.810032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" event={"ID":"bda3d049-a122-4123-98c2-cbfa5b51f4ba","Type":"ContainerStarted","Data":"2aa32da1909d5274bf6f365810becfc6aaa00e461d5da1d293d26a46fc9fcfef"}
Jul 15 11:08:46.833799 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:46.833752    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" podStartSLOduration=1.833737917 podStartE2EDuration="1.833737917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:08:46.83082977 +0000 UTC m=+1392.665574221" watchObservedRunningTime="2026-07-15 11:08:46.833737917 +0000 UTC m=+1392.668482366"
Jul 15 11:08:47.221029 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:47.220985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:47.238389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:47.238361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:47.815651 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:47.815592    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" podUID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" containerName="istio-proxy" containerID="cri-o://c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da" gracePeriod=30
Jul 15 11:08:47.815871 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:47.815728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:47.817372 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:47.817349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:53.078888 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.078820    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:53.188961 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.188930    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.188971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz2c9\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189460 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189460 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy\") pod \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\" (UID: \"bda3d049-a122-4123-98c2-cbfa5b51f4ba\") "
Jul 15 11:08:53.189578 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189464    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:53.189578 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:08:53.189578 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189554    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:53.189728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:53.191727 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.189780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:53.196088 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.193196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:08:53.197525 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.197456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9" (OuterVolumeSpecName: "kube-api-access-bz2c9") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "kube-api-access-bz2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:08:53.197622 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.197491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 11:08:53.198784 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.198763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bda3d049-a122-4123-98c2-cbfa5b51f4ba" (UID: "bda3d049-a122-4123-98c2-cbfa5b51f4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:53.290236 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290197    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istiod-ca-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290236 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290231    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2c9\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-kube-api-access-bz2c9\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290236 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290242    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-credential-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290252    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-data\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290260    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290268    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-workload-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290275    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290282    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-envoy\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.290459 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.290289    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda3d049-a122-4123-98c2-cbfa5b51f4ba-istio-podinfo\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:08:53.851337 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.851300    2565 generic.go:356] "Generic (PLEG): container finished" podID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" containerID="c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da" exitCode=0
Jul 15 11:08:53.851488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.851378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"
Jul 15 11:08:53.851488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.851389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" event={"ID":"bda3d049-a122-4123-98c2-cbfa5b51f4ba","Type":"ContainerDied","Data":"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"}
Jul 15 11:08:53.851488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.851449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687" event={"ID":"bda3d049-a122-4123-98c2-cbfa5b51f4ba","Type":"ContainerDied","Data":"2aa32da1909d5274bf6f365810becfc6aaa00e461d5da1d293d26a46fc9fcfef"}
Jul 15 11:08:53.851488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.851471    2565 scope.go:117] "RemoveContainer" containerID="c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"
Jul 15 11:08:53.863148 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.863128    2565 scope.go:117] "RemoveContainer" containerID="c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"
Jul 15 11:08:53.863556 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:08:53.863434    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da\": container with ID starting with c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da not found: ID does not exist" containerID="c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"
Jul 15 11:08:53.863556 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.863462    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da"} err="failed to get container status \"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da\": rpc error: code = NotFound desc = could not find container \"c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da\": container with ID starting with c701823d8421970ef6ba654d66ca8410fbf534f81b37344160ecefea6f1071da not found: ID does not exist"
Jul 15 11:08:53.878564 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.878529    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:53.883064 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:53.883025    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687"]
Jul 15 11:08:54.673853 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:08:54.673814    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" path="/var/lib/kubelet/pods/bda3d049-a122-4123-98c2-cbfa5b51f4ba/volumes"
Jul 15 11:09:07.883814 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:07.883779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:09:19.497369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:19.497333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:09:30.217723 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:30.217691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:09:37.582266 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:37.582230    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:09:37.589095 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:37.589063    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:09:37.654580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:37.654544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:09:39.367720 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:39.367680    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:09:49.368136 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:49.368092    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:09:52.604358 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.604299    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main" containerID="cri-o://082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6" gracePeriod=45
Jul 15 11:09:52.868517 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.868455    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:09:52.912163 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.912133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds\") pod \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") "
Jul 15 11:09:52.912338 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.912234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs\") pod \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") "
Jul 15 11:09:52.912338 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.912269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqjld\" (UniqueName: \"kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld\") pod \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") "
Jul 15 11:09:52.912338 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.912295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location\") pod \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\" (UID: \"0edb146b-c10e-461d-9c77-ddf0bfa148d7\") "
Jul 15 11:09:52.912510 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.912481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0edb146b-c10e-461d-9c77-ddf0bfa148d7" (UID: "0edb146b-c10e-461d-9c77-ddf0bfa148d7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:52.913093 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.913030    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0edb146b-c10e-461d-9c77-ddf0bfa148d7" (UID: "0edb146b-c10e-461d-9c77-ddf0bfa148d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:52.914515 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.914494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0edb146b-c10e-461d-9c77-ddf0bfa148d7" (UID: "0edb146b-c10e-461d-9c77-ddf0bfa148d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:09:52.914621 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:52.914568    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld" (OuterVolumeSpecName: "kube-api-access-tqjld") pod "0edb146b-c10e-461d-9c77-ddf0bfa148d7" (UID: "0edb146b-c10e-461d-9c77-ddf0bfa148d7"). InnerVolumeSpecName "kube-api-access-tqjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:09:53.013702 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.013666    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.013702 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.013696    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjld\" (UniqueName: \"kubernetes.io/projected/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kube-api-access-tqjld\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.013702 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.013706    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.013945 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.013715    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0edb146b-c10e-461d-9c77-ddf0bfa148d7-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.179870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.179781    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerID="082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6" exitCode=0
Jul 15 11:09:53.179870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.179828    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"
Jul 15 11:09:53.180081 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.179828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerDied","Data":"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"}
Jul 15 11:09:53.180081 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.179957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9" event={"ID":"0edb146b-c10e-461d-9c77-ddf0bfa148d7","Type":"ContainerDied","Data":"5ea53083cd16ca5a253f1ec24b93922b2398f6df7e880b2932bea57aa6d661af"}
Jul 15 11:09:53.180081 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.179985    2565 scope.go:117] "RemoveContainer" containerID="082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"
Jul 15 11:09:53.189468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.189451    2565 scope.go:117] "RemoveContainer" containerID="b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"
Jul 15 11:09:53.197877 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.197857    2565 scope.go:117] "RemoveContainer" containerID="082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"
Jul 15 11:09:53.198131 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:09:53.198111    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6\": container with ID starting with 082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6 not found: ID does not exist" containerID="082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"
Jul 15 11:09:53.198183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.198139    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6"} err="failed to get container status \"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6\": rpc error: code = NotFound desc = could not find container \"082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6\": container with ID starting with 082ebe62e839d43c1b23e3f5dcebaf596595b1e324eb6874500c5542123c56f6 not found: ID does not exist"
Jul 15 11:09:53.198183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.198157    2565 scope.go:117] "RemoveContainer" containerID="b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"
Jul 15 11:09:53.198401 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:09:53.198382    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb\": container with ID starting with b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb not found: ID does not exist" containerID="b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"
Jul 15 11:09:53.198444 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.198409    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb"} err="failed to get container status \"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb\": rpc error: code = NotFound desc = could not find container \"b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb\": container with ID starting with b3576de015b04a099b00fa419f69ff05b467159ad72e871c2880ab762101d8bb not found: ID does not exist"
Jul 15 11:09:53.202212 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.202188    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:09:53.208513 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:53.208491    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9"]
Jul 15 11:09:54.674804 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:09:54.674773    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" path="/var/lib/kubelet/pods/0edb146b-c10e-461d-9c77-ddf0bfa148d7/volumes"
Jul 15 11:10:03.958254 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958219    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958599    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" containerName="istio-proxy"
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958610    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" containerName="istio-proxy"
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958620    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main"
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main"
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958635    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="storage-initializer"
Jul 15 11:10:03.958659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958642    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="storage-initializer"
Jul 15 11:10:03.958881 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edb146b-c10e-461d-9c77-ddf0bfa148d7" containerName="main"
Jul 15 11:10:03.958881 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.958717    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda3d049-a122-4123-98c2-cbfa5b51f4ba" containerName="istio-proxy"
Jul 15 11:10:03.961757 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.961741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:03.964311 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.964285    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-jqm47\""
Jul 15 11:10:03.965119 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.965096    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 15 11:10:03.965119 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.965116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:10:03.965323 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.965180    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:10:03.965591 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.965568    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:10:03.965729 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.965711    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:10:03.973154 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:03.973131    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:10:04.114283 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.114250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.114491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.114292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.114491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.114311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.114491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.114332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmrx\" (UniqueName: \"kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.214956 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.214856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.214956 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.214916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.214956 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.214944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.215213 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.214971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmrx\" (UniqueName: \"kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.215301 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.215281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.215362 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.215338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.217584 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.217562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.224564 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.224518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmrx\" (UniqueName: \"kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.249860 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.249817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:04.272075 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.272023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:04.350285 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.350253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:04.413742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:04.413716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:10:04.415388 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:10:04.415354    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d90b98f_e304_408f_a3bb_4b5723bba361.slice/crio-e9a3fd3d354faf3236d19533a26e7b12fa43795fcc6095723427ac02a142aa54 WatchSource:0}: Error finding container e9a3fd3d354faf3236d19533a26e7b12fa43795fcc6095723427ac02a142aa54: Status 404 returned error can't find the container with id e9a3fd3d354faf3236d19533a26e7b12fa43795fcc6095723427ac02a142aa54
Jul 15 11:10:05.249694 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:05.249662    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerID="55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb" exitCode=0
Jul 15 11:10:05.250105 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:05.249732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" event={"ID":"9d90b98f-e304-408f-a3bb-4b5723bba361","Type":"ContainerDied","Data":"55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb"}
Jul 15 11:10:05.250105 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:05.249772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" event={"ID":"9d90b98f-e304-408f-a3bb-4b5723bba361","Type":"ContainerStarted","Data":"e9a3fd3d354faf3236d19533a26e7b12fa43795fcc6095723427ac02a142aa54"}
Jul 15 11:10:06.257492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:06.257457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" event={"ID":"9d90b98f-e304-408f-a3bb-4b5723bba361","Type":"ContainerStarted","Data":"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"}
Jul 15 11:10:06.257870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:06.257531    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:06.296145 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:06.296087    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" podStartSLOduration=2.6312234500000002 podStartE2EDuration="3.2960723s" totalImagesPullingTime="0s" totalInitContainerRuntime="664.84885ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:10:06.291939573 +0000 UTC m=+1472.126684023" watchObservedRunningTime="2026-07-15 11:10:06.2960723 +0000 UTC m=+1472.130816750"
Jul 15 11:10:08.685228 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.685193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:08.883098 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.883065    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:10:08.887258 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.887239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:08.889716 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.889695    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 15 11:10:08.889817 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.889750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-98sjw\""
Jul 15 11:10:08.896492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.896470    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:10:08.903297 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.903272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:08.955947 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.955855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:08.955947 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.955912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:08.956183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.955986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:08.956183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:08.956033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsps\" (UniqueName: \"kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.056628 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.056593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.056797 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.056642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.056870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.056786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.056870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.056827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsps\" (UniqueName: \"kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.057004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.056960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.057085 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.057023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.059603 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.059567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.066325 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.066302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsps\" (UniqueName: \"kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.197309 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.197273    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:09.345558 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:09.345530    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:10:09.347937 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:10:09.347909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc862a7fb_546a_4955_87b4_c7a41d12b218.slice/crio-5a0f43daa834414c93d5abea1535240494898fe46dafd4c001c716c0abe086ef WatchSource:0}: Error finding container 5a0f43daa834414c93d5abea1535240494898fe46dafd4c001c716c0abe086ef: Status 404 returned error can't find the container with id 5a0f43daa834414c93d5abea1535240494898fe46dafd4c001c716c0abe086ef
Jul 15 11:10:10.282436 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:10.282395    2565 generic.go:356] "Generic (PLEG): container finished" podID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerID="a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0" exitCode=0
Jul 15 11:10:10.282832 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:10.282474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" event={"ID":"c862a7fb-546a-4955-87b4-c7a41d12b218","Type":"ContainerDied","Data":"a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0"}
Jul 15 11:10:10.282832 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:10.282520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" event={"ID":"c862a7fb-546a-4955-87b4-c7a41d12b218","Type":"ContainerStarted","Data":"5a0f43daa834414c93d5abea1535240494898fe46dafd4c001c716c0abe086ef"}
Jul 15 11:10:11.290930 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:11.290891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" event={"ID":"c862a7fb-546a-4955-87b4-c7a41d12b218","Type":"ContainerStarted","Data":"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"}
Jul 15 11:10:11.291428 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:11.291003    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:10:11.317447 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:11.317398    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" podStartSLOduration=2.672203915 podStartE2EDuration="3.317384755s" totalImagesPullingTime="0s" totalInitContainerRuntime="645.18084ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:10:11.315458131 +0000 UTC m=+1477.150202598" watchObservedRunningTime="2026-07-15 11:10:11.317384755 +0000 UTC m=+1477.152129204"
Jul 15 11:10:16.088706 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:16.088660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:16.972744 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:16.972710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:27.631771 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:27.631733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:34.691849 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:34.691816    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:10:34.693519 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:34.693491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:10:34.695933 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:34.695915    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:10:34.697215 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:34.697193    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:10:37.264442 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:37.264410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:10:38.954584 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:38.954546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:10:42.298512 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:10:42.298481    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:11:53.662240 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:11:53.662201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:18.624786 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:18.624746    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:12:18.649471 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:18.649433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:18.743257 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:18.743222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:26.477756 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:26.477680    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:27.264875 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:27.264829    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:12:33.646636 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:33.646596    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="main" containerID="cri-o://d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e" gracePeriod=45
Jul 15 11:12:33.894556 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:33.894532    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:12:34.044938 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.044862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds\") pod \"9d90b98f-e304-408f-a3bb-4b5723bba361\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") "
Jul 15 11:12:34.044938 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.044921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs\") pod \"9d90b98f-e304-408f-a3bb-4b5723bba361\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") "
Jul 15 11:12:34.045208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.044959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhmrx\" (UniqueName: \"kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx\") pod \"9d90b98f-e304-408f-a3bb-4b5723bba361\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") "
Jul 15 11:12:34.045208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.044985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location\") pod \"9d90b98f-e304-408f-a3bb-4b5723bba361\" (UID: \"9d90b98f-e304-408f-a3bb-4b5723bba361\") "
Jul 15 11:12:34.045208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.045160    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9d90b98f-e304-408f-a3bb-4b5723bba361" (UID: "9d90b98f-e304-408f-a3bb-4b5723bba361"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.045372 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.045269    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.045642 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.045614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d90b98f-e304-408f-a3bb-4b5723bba361" (UID: "9d90b98f-e304-408f-a3bb-4b5723bba361"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.047353 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.047328    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d90b98f-e304-408f-a3bb-4b5723bba361" (UID: "9d90b98f-e304-408f-a3bb-4b5723bba361"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:12:34.047450 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.047394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx" (OuterVolumeSpecName: "kube-api-access-lhmrx") pod "9d90b98f-e304-408f-a3bb-4b5723bba361" (UID: "9d90b98f-e304-408f-a3bb-4b5723bba361"). InnerVolumeSpecName "kube-api-access-lhmrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:12:34.066344 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.066315    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerID="d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e" exitCode=0
Jul 15 11:12:34.066468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.066364    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"
Jul 15 11:12:34.066468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.066404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" event={"ID":"9d90b98f-e304-408f-a3bb-4b5723bba361","Type":"ContainerDied","Data":"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"}
Jul 15 11:12:34.066468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.066459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh" event={"ID":"9d90b98f-e304-408f-a3bb-4b5723bba361","Type":"ContainerDied","Data":"e9a3fd3d354faf3236d19533a26e7b12fa43795fcc6095723427ac02a142aa54"}
Jul 15 11:12:34.066572 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.066474    2565 scope.go:117] "RemoveContainer" containerID="d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"
Jul 15 11:12:34.075550 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.075531    2565 scope.go:117] "RemoveContainer" containerID="55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb"
Jul 15 11:12:34.083906 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.083883    2565 scope.go:117] "RemoveContainer" containerID="d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"
Jul 15 11:12:34.084251 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:12:34.084231    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e\": container with ID starting with d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e not found: ID does not exist" containerID="d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"
Jul 15 11:12:34.084336 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.084262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e"} err="failed to get container status \"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e\": rpc error: code = NotFound desc = could not find container \"d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e\": container with ID starting with d47230a9131df30bb8bbe2b84d52af59c9e678f3249ef447aeb1962376292a6e not found: ID does not exist"
Jul 15 11:12:34.084336 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.084288    2565 scope.go:117] "RemoveContainer" containerID="55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb"
Jul 15 11:12:34.084516 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:12:34.084498    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb\": container with ID starting with 55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb not found: ID does not exist" containerID="55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb"
Jul 15 11:12:34.084557 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.084524    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb"} err="failed to get container status \"55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb\": rpc error: code = NotFound desc = could not find container \"55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb\": container with ID starting with 55db8c3ba19e40cc163e6fe5a25d2f84682c277659b94f85e0a09c9b080eaaeb not found: ID does not exist"
Jul 15 11:12:34.088605 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.088579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:12:34.094589 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.094566    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh"]
Jul 15 11:12:34.145660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.145635    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d90b98f-e304-408f-a3bb-4b5723bba361-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.145660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.145658    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d90b98f-e304-408f-a3bb-4b5723bba361-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.145847 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.145670    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmrx\" (UniqueName: \"kubernetes.io/projected/9d90b98f-e304-408f-a3bb-4b5723bba361-kube-api-access-lhmrx\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.681906 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:34.681866    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" path="/var/lib/kubelet/pods/9d90b98f-e304-408f-a3bb-4b5723bba361/volumes"
Jul 15 11:12:42.466646 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.466612    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:42.794741 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.794654    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:12:42.795058 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.795028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="storage-initializer"
Jul 15 11:12:42.795058 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.795060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="storage-initializer"
Jul 15 11:12:42.795219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.795068    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="main"
Jul 15 11:12:42.795219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.795073    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="main"
Jul 15 11:12:42.795219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.795144    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d90b98f-e304-408f-a3bb-4b5723bba361" containerName="main"
Jul 15 11:12:42.799946 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.799926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.803192 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.803167    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-s6flf\""
Jul 15 11:12:42.803956 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.803939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 15 11:12:42.816849 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.816825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:12:42.818455 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.818431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.818540 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.818470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.818540 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.818499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.818540 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.818515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6f9\" (UniqueName: \"kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.832132 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.832091    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:42.919030 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.918994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.919234 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.919069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.919234 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.919103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.919234 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.919119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6f9\" (UniqueName: \"kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.919488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.919463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.919550 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.919475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.921689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.921669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:42.928817 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:42.928791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6f9\" (UniqueName: \"kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:43.110051 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:43.110016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:43.254806 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:43.254781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:12:43.256462 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:12:43.256433    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eba364b_30d8_4587_8f28_4d85b6dbc7cb.slice/crio-bc5864c133758925129eeb1c69d018393bde836159ff07261b1433c22a79c925 WatchSource:0}: Error finding container bc5864c133758925129eeb1c69d018393bde836159ff07261b1433c22a79c925: Status 404 returned error can't find the container with id bc5864c133758925129eeb1c69d018393bde836159ff07261b1433c22a79c925
Jul 15 11:12:44.127240 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:44.127204    2565 generic.go:356] "Generic (PLEG): container finished" podID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerID="d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039" exitCode=0
Jul 15 11:12:44.127751 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:44.127279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" event={"ID":"4eba364b-30d8-4587-8f28-4d85b6dbc7cb","Type":"ContainerDied","Data":"d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039"}
Jul 15 11:12:44.127751 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:44.127342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" event={"ID":"4eba364b-30d8-4587-8f28-4d85b6dbc7cb","Type":"ContainerStarted","Data":"bc5864c133758925129eeb1c69d018393bde836159ff07261b1433c22a79c925"}
Jul 15 11:12:45.136235 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:45.136204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" event={"ID":"4eba364b-30d8-4587-8f28-4d85b6dbc7cb","Type":"ContainerStarted","Data":"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"}
Jul 15 11:12:45.136638 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:45.136337    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:12:45.158191 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:45.158139    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" podStartSLOduration=2.515683525 podStartE2EDuration="3.158118092s" totalImagesPullingTime="0s" totalInitContainerRuntime="642.434567ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:12:45.154848922 +0000 UTC m=+1630.989593371" watchObservedRunningTime="2026-07-15 11:12:45.158118092 +0000 UTC m=+1630.992862567"
Jul 15 11:12:49.427487 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:49.427452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:12:53.028650 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:12:53.028617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:13:01.400181 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:13:01.400151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:13:16.143487 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:13:16.143452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:13:57.959898 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:13:57.959817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:14:48.893433 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:14:48.893393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:14:48.908341 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:14:48.908307    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:14:48.983253 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:14:48.983213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:14:56.143716 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:14:56.143673    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:15:03.938926 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:03.938857    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="main" containerID="cri-o://b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e" gracePeriod=45
Jul 15 11:15:04.204814 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.204789    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:15:04.345873 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.345837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv6f9\" (UniqueName: \"kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9\") pod \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") "
Jul 15 11:15:04.346096 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.345920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs\") pod \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") "
Jul 15 11:15:04.346096 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.345964    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds\") pod \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") "
Jul 15 11:15:04.346096 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.346011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location\") pod \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\" (UID: \"4eba364b-30d8-4587-8f28-4d85b6dbc7cb\") "
Jul 15 11:15:04.346307 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.346269    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4eba364b-30d8-4587-8f28-4d85b6dbc7cb" (UID: "4eba364b-30d8-4587-8f28-4d85b6dbc7cb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.346713 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.346693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4eba364b-30d8-4587-8f28-4d85b6dbc7cb" (UID: "4eba364b-30d8-4587-8f28-4d85b6dbc7cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.348165 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.348142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9" (OuterVolumeSpecName: "kube-api-access-rv6f9") pod "4eba364b-30d8-4587-8f28-4d85b6dbc7cb" (UID: "4eba364b-30d8-4587-8f28-4d85b6dbc7cb"). InnerVolumeSpecName "kube-api-access-rv6f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:15:04.348165 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.348150    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eba364b-30d8-4587-8f28-4d85b6dbc7cb" (UID: "4eba364b-30d8-4587-8f28-4d85b6dbc7cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:15:04.447592 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.447503    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.447592 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.447534    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.447592 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.447544    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.447592 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.447552    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6f9\" (UniqueName: \"kubernetes.io/projected/4eba364b-30d8-4587-8f28-4d85b6dbc7cb-kube-api-access-rv6f9\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.929546 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.929514    2565 generic.go:356] "Generic (PLEG): container finished" podID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerID="b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e" exitCode=0
Jul 15 11:15:04.929775 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.929567    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"
Jul 15 11:15:04.929775 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.929602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" event={"ID":"4eba364b-30d8-4587-8f28-4d85b6dbc7cb","Type":"ContainerDied","Data":"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"}
Jul 15 11:15:04.929775 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.929677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb" event={"ID":"4eba364b-30d8-4587-8f28-4d85b6dbc7cb","Type":"ContainerDied","Data":"bc5864c133758925129eeb1c69d018393bde836159ff07261b1433c22a79c925"}
Jul 15 11:15:04.929775 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.929706    2565 scope.go:117] "RemoveContainer" containerID="b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"
Jul 15 11:15:04.940007 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.939818    2565 scope.go:117] "RemoveContainer" containerID="d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039"
Jul 15 11:15:04.949473 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.949451    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:15:04.949794 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.949780    2565 scope.go:117] "RemoveContainer" containerID="b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"
Jul 15 11:15:04.950139 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:15:04.950110    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e\": container with ID starting with b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e not found: ID does not exist" containerID="b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"
Jul 15 11:15:04.950250 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.950149    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e"} err="failed to get container status \"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e\": rpc error: code = NotFound desc = could not find container \"b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e\": container with ID starting with b3d0a7f6a1e83463daf8491564e0f11b5666f05e00c623f50c9ef9faf3d2336e not found: ID does not exist"
Jul 15 11:15:04.950250 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.950174    2565 scope.go:117] "RemoveContainer" containerID="d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039"
Jul 15 11:15:04.950470 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:15:04.950454    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039\": container with ID starting with d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039 not found: ID does not exist" containerID="d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039"
Jul 15 11:15:04.950515 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.950475    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039"} err="failed to get container status \"d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039\": rpc error: code = NotFound desc = could not find container \"d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039\": container with ID starting with d47034d251021d8c1e640db102c6b0f40ba5f40d514047768451b2de862dc039 not found: ID does not exist"
Jul 15 11:15:04.951634 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:04.951616    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb"]
Jul 15 11:15:06.674279 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:06.674245    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" path="/var/lib/kubelet/pods/4eba364b-30d8-4587-8f28-4d85b6dbc7cb/volumes"
Jul 15 11:15:09.442160 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442122    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:09.442623 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442605    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="main"
Jul 15 11:15:09.442667 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="main"
Jul 15 11:15:09.442667 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442646    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="storage-initializer"
Jul 15 11:15:09.442667 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442651    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="storage-initializer"
Jul 15 11:15:09.442759 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.442717    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eba364b-30d8-4587-8f28-4d85b6dbc7cb" containerName="main"
Jul 15 11:15:09.447367 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.447344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.449907 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.449884    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 11:15:09.450228 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.449892    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-95m52\""
Jul 15 11:15:09.459666 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.459639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:09.593096 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfh4\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.593465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.593435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694075 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.693975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694075 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694075 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694360 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694360 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694360 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694360 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfh4\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694714 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694800 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.694841 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.694809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.696824 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.696798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.697019 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.697002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.711675 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.711638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfh4\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.732406 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.732381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-f2zbm\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.760033 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.760006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:09.910675 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.910649    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:09.912554 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:15:09.912510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5976eb4d_07e4_4965_af4e_486e5dfe73be.slice/crio-bd4a2aa6234b2ad90c6949fd77dbaa9bae49a5f0ce33a37930fa3a225cf4c2a1 WatchSource:0}: Error finding container bd4a2aa6234b2ad90c6949fd77dbaa9bae49a5f0ce33a37930fa3a225cf4c2a1: Status 404 returned error can't find the container with id bd4a2aa6234b2ad90c6949fd77dbaa9bae49a5f0ce33a37930fa3a225cf4c2a1
Jul 15 11:15:09.914522 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.914485    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:15:09.914629 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.914549    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:15:09.914629 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.914575    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:15:09.960227 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:09.960193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" event={"ID":"5976eb4d-07e4-4965-af4e-486e5dfe73be","Type":"ContainerStarted","Data":"bd4a2aa6234b2ad90c6949fd77dbaa9bae49a5f0ce33a37930fa3a225cf4c2a1"}
Jul 15 11:15:10.966917 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:10.966878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" event={"ID":"5976eb4d-07e4-4965-af4e-486e5dfe73be","Type":"ContainerStarted","Data":"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"}
Jul 15 11:15:10.990194 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:10.990140    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" podStartSLOduration=1.990124792 podStartE2EDuration="1.990124792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:15:10.987599412 +0000 UTC m=+1776.822343903" watchObservedRunningTime="2026-07-15 11:15:10.990124792 +0000 UTC m=+1776.824869242"
Jul 15 11:15:11.399662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:11.399622    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:11.760431 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:11.760349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:11.765770 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:11.765744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:11.972135 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:11.972098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:11.973023 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:11.973003    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:12.976762 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:12.976716    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" podUID="5976eb4d-07e4-4965-af4e-486e5dfe73be" containerName="istio-proxy" containerID="cri-o://14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037" gracePeriod=30
Jul 15 11:15:16.273145 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:16.273098    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:15:17.142580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.142544    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:15:17.146305 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.146283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.151228 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.151203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 15 11:15:17.151713 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.151509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-6fxdd\""
Jul 15 11:15:17.167925 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.167884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:15:17.270809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.270526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.270809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.270584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.270809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.270624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gllf\" (UniqueName: \"kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.270809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.270694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.285841 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.285811    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:15:17.372134 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.372098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.372489 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.372464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.372619 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.372602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.372761 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.372744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gllf\" (UniqueName: \"kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.373703 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.373679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.373871 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.373714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.376034 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.376014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:15:17.376547 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.376527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.382300 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.382281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gllf\" (UniqueName: \"kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf\") pod \"router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.462513 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.462428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:17.639029 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:17.639003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:15:17.640830 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:15:17.640792    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c5d3e1_53e6_4f37_8940_3d11bef1fbcd.slice/crio-27c6db25706f196a7e3da98ad3c08aa4ec168dd95fd8c7c111e1823618b68ab2 WatchSource:0}: Error finding container 27c6db25706f196a7e3da98ad3c08aa4ec168dd95fd8c7c111e1823618b68ab2: Status 404 returned error can't find the container with id 27c6db25706f196a7e3da98ad3c08aa4ec168dd95fd8c7c111e1823618b68ab2
Jul 15 11:15:18.002923 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.002827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerStarted","Data":"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"}
Jul 15 11:15:18.002923 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.002890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerStarted","Data":"27c6db25706f196a7e3da98ad3c08aa4ec168dd95fd8c7c111e1823618b68ab2"}
Jul 15 11:15:18.222837 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.222810    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:18.385027 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.384994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgfh4\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385454 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385448    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385847 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385847 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket\") pod \"5976eb4d-07e4-4965-af4e-486e5dfe73be\" (UID: \"5976eb4d-07e4-4965-af4e-486e5dfe73be\") "
Jul 15 11:15:18.385847 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data" (OuterVolumeSpecName: "istio-data") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:18.386004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385840    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-data\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.386004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385876    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:18.386004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385892    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.386004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:18.386004 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.385994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:15:18.389163 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.389110    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:18.389163 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.389132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token" (OuterVolumeSpecName: "istio-token") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:15:18.390732 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.390703    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4" (OuterVolumeSpecName: "kube-api-access-vgfh4") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "kube-api-access-vgfh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:15:18.390838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.390792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5976eb4d-07e4-4965-af4e-486e5dfe73be" (UID: "5976eb4d-07e4-4965-af4e-486e5dfe73be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 11:15:18.486497 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486463    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-envoy\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486497 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486492    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfh4\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-kube-api-access-vgfh4\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486497 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486502    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5976eb4d-07e4-4965-af4e-486e5dfe73be-istiod-ca-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486514    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-podinfo\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486527    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-credential-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486538    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5976eb4d-07e4-4965-af4e-486e5dfe73be-istio-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:18.486728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:18.486546    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5976eb4d-07e4-4965-af4e-486e5dfe73be-workload-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:15:19.009167 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.009087    2565 generic.go:356] "Generic (PLEG): container finished" podID="5976eb4d-07e4-4965-af4e-486e5dfe73be" containerID="14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037" exitCode=0
Jul 15 11:15:19.009167 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.009135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"
Jul 15 11:15:19.009401 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.009172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" event={"ID":"5976eb4d-07e4-4965-af4e-486e5dfe73be","Type":"ContainerDied","Data":"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"}
Jul 15 11:15:19.009401 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.009221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm" event={"ID":"5976eb4d-07e4-4965-af4e-486e5dfe73be","Type":"ContainerDied","Data":"bd4a2aa6234b2ad90c6949fd77dbaa9bae49a5f0ce33a37930fa3a225cf4c2a1"}
Jul 15 11:15:19.009401 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.009244    2565 scope.go:117] "RemoveContainer" containerID="14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"
Jul 15 11:15:19.010840 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.010794    2565 generic.go:356] "Generic (PLEG): container finished" podID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerID="479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998" exitCode=0
Jul 15 11:15:19.010967 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.010848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerDied","Data":"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"}
Jul 15 11:15:19.021310 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.021286    2565 scope.go:117] "RemoveContainer" containerID="14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"
Jul 15 11:15:19.021902 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:15:19.021884    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037\": container with ID starting with 14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037 not found: ID does not exist" containerID="14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"
Jul 15 11:15:19.021982 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.021909    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037"} err="failed to get container status \"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037\": rpc error: code = NotFound desc = could not find container \"14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037\": container with ID starting with 14cd32819abde427a8742ec4f187d3c11b840264b0b9016ff529d5e7572e9037 not found: ID does not exist"
Jul 15 11:15:19.031328 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.031303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:19.036085 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:19.036060    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm"]
Jul 15 11:15:20.018138 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:20.018101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerStarted","Data":"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"}
Jul 15 11:15:20.018582 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:20.018205    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:15:20.041933 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:20.041870    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" podStartSLOduration=2.335491187 podStartE2EDuration="3.04185596s" totalImagesPullingTime="0s" totalInitContainerRuntime="706.364773ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:15:20.038959321 +0000 UTC m=+1785.873703800" watchObservedRunningTime="2026-07-15 11:15:20.04185596 +0000 UTC m=+1785.876600409"
Jul 15 11:15:20.674136 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:20.674101    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5976eb4d-07e4-4965-af4e-486e5dfe73be" path="/var/lib/kubelet/pods/5976eb4d-07e4-4965-af4e-486e5dfe73be/volumes"
Jul 15 11:15:34.733099 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:34.733072    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:15:34.737382 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:34.737358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:15:34.738265 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:34.738239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:15:34.742359 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:34.742340    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:15:51.026609 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:15:51.026575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:17:21.993613 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:21.993576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:17:22.038214 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:22.038176    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:17:24.949650 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:24.949610    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:17:31.026737 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:31.026699    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:17:37.014746 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.014703    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="main" containerID="cri-o://70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97" gracePeriod=45
Jul 15 11:17:37.273260 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.273199    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:17:37.416957 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.416920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds\") pod \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") "
Jul 15 11:17:37.417170 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417020    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location\") pod \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") "
Jul 15 11:17:37.417170 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs\") pod \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") "
Jul 15 11:17:37.417170 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417161    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gllf\" (UniqueName: \"kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf\") pod \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\" (UID: \"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd\") "
Jul 15 11:17:37.417311 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" (UID: "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.417408 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417396    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.417841 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.417719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" (UID: "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.419389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.419362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" (UID: "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:37.419389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.419373    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf" (OuterVolumeSpecName: "kube-api-access-5gllf") pod "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" (UID: "10c5d3e1-53e6-4f37-8940-3d11bef1fbcd"). InnerVolumeSpecName "kube-api-access-5gllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:37.518669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.518641    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gllf\" (UniqueName: \"kubernetes.io/projected/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kube-api-access-5gllf\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.518669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.518664    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.518669 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.518675    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.774386 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.774346    2565 generic.go:356] "Generic (PLEG): container finished" podID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerID="70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97" exitCode=0
Jul 15 11:17:37.774555 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.774448    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"
Jul 15 11:17:37.774555 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.774481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerDied","Data":"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"}
Jul 15 11:17:37.774555 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.774535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8" event={"ID":"10c5d3e1-53e6-4f37-8940-3d11bef1fbcd","Type":"ContainerDied","Data":"27c6db25706f196a7e3da98ad3c08aa4ec168dd95fd8c7c111e1823618b68ab2"}
Jul 15 11:17:37.774658 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.774558    2565 scope.go:117] "RemoveContainer" containerID="70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"
Jul 15 11:17:37.784654 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.784636    2565 scope.go:117] "RemoveContainer" containerID="479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"
Jul 15 11:17:37.793368 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.793350    2565 scope.go:117] "RemoveContainer" containerID="70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"
Jul 15 11:17:37.793669 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:17:37.793651    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97\": container with ID starting with 70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97 not found: ID does not exist" containerID="70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"
Jul 15 11:17:37.793727 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.793679    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97"} err="failed to get container status \"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97\": rpc error: code = NotFound desc = could not find container \"70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97\": container with ID starting with 70eb60566e77c267053cfaa87ed387df7e751e0c148b29511091f05b2e9e7a97 not found: ID does not exist"
Jul 15 11:17:37.793727 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.793703    2565 scope.go:117] "RemoveContainer" containerID="479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"
Jul 15 11:17:37.793964 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:17:37.793948    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998\": container with ID starting with 479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998 not found: ID does not exist" containerID="479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"
Jul 15 11:17:37.794015 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.793976    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998"} err="failed to get container status \"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998\": rpc error: code = NotFound desc = could not find container \"479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998\": container with ID starting with 479a2b72db9cfc70395e5a0e10c25c3e27c4d1501945fb0b5d721490df7ae998 not found: ID does not exist"
Jul 15 11:17:37.798608 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.798585    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:17:37.802577 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:37.802556    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8"]
Jul 15 11:17:38.673820 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:38.673787    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" path="/var/lib/kubelet/pods/10c5d3e1-53e6-4f37-8940-3d11bef1fbcd/volumes"
Jul 15 11:17:54.470521 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.470479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:17:54.749976 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.749891    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:17:54.750325 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750308    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5976eb4d-07e4-4965-af4e-486e5dfe73be" containerName="istio-proxy"
Jul 15 11:17:54.750325 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750325    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5976eb4d-07e4-4965-af4e-486e5dfe73be" containerName="istio-proxy"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750339    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="storage-initializer"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750345    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="storage-initializer"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="main"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="main"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750435    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5976eb4d-07e4-4965-af4e-486e5dfe73be" containerName="istio-proxy"
Jul 15 11:17:54.750464 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.750442    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c5d3e1-53e6-4f37-8940-3d11bef1fbcd" containerName="main"
Jul 15 11:17:54.755091 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.755071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.757861 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.757836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 15 11:17:54.757991 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.757864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-kgq2z\""
Jul 15 11:17:54.772406 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.772382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:17:54.787014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.786977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:17:54.874196 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.874163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cjx\" (UniqueName: \"kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.874381 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.874205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.874381 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.874228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.874381 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.874247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.975689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.975646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cjx\" (UniqueName: \"kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.975689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.975699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.975969 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.975814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.975969 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.975853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.976117 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.976023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.976169 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.976145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.978457 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.978434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:54.985073 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:54.985033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cjx\" (UniqueName: \"kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:55.064779 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:55.064692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:55.216036 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:55.215998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:17:55.217609 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:17:55.217579    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be114d7_1447_4eea_a823_029e5483452d.slice/crio-333fe309d63c9a00ea0c0d56ea9b5a29c9128c4bb285363aa8b43d87329297e1 WatchSource:0}: Error finding container 333fe309d63c9a00ea0c0d56ea9b5a29c9128c4bb285363aa8b43d87329297e1: Status 404 returned error can't find the container with id 333fe309d63c9a00ea0c0d56ea9b5a29c9128c4bb285363aa8b43d87329297e1
Jul 15 11:17:55.873625 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:55.873585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerStarted","Data":"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"}
Jul 15 11:17:55.874006 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:55.873637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerStarted","Data":"333fe309d63c9a00ea0c0d56ea9b5a29c9128c4bb285363aa8b43d87329297e1"}
Jul 15 11:17:56.880484 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:56.880448    2565 generic.go:356] "Generic (PLEG): container finished" podID="8be114d7-1447-4eea-a823-029e5483452d" containerID="f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07" exitCode=0
Jul 15 11:17:56.880868 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:56.880544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerDied","Data":"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"}
Jul 15 11:17:57.874797 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.874758    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:17:57.890254 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.890204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerStarted","Data":"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"}
Jul 15 11:17:57.890710 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.890469    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:17:57.893848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.893828    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:17:57.917784 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.917741    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" podStartSLOduration=3.256698163 podStartE2EDuration="3.91772559s" totalImagesPullingTime="0s" totalInitContainerRuntime="661.027427ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:57.916465424 +0000 UTC m=+1943.751209895" watchObservedRunningTime="2026-07-15 11:17:57.91772559 +0000 UTC m=+1943.752470040"
Jul 15 11:17:57.974472 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:17:57.974433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:02.298022 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:02.297985    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:18:12.240910 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:12.240869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:12.297985 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:12.297948    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:18:12.914187 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:12.914131    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main" containerID="cri-o://595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd" gracePeriod=45
Jul 15 11:18:13.165588 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.165529    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:18:13.226663 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.226633    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs\") pod \"c862a7fb-546a-4955-87b4-c7a41d12b218\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") "
Jul 15 11:18:13.226808 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.226676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fsps\" (UniqueName: \"kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps\") pod \"c862a7fb-546a-4955-87b4-c7a41d12b218\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") "
Jul 15 11:18:13.226851 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.226803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location\") pod \"c862a7fb-546a-4955-87b4-c7a41d12b218\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") "
Jul 15 11:18:13.226886 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.226868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds\") pod \"c862a7fb-546a-4955-87b4-c7a41d12b218\" (UID: \"c862a7fb-546a-4955-87b4-c7a41d12b218\") "
Jul 15 11:18:13.227144 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.227116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c862a7fb-546a-4955-87b4-c7a41d12b218" (UID: "c862a7fb-546a-4955-87b4-c7a41d12b218"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.227552 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.227530    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c862a7fb-546a-4955-87b4-c7a41d12b218" (UID: "c862a7fb-546a-4955-87b4-c7a41d12b218"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.228977 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.228954    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c862a7fb-546a-4955-87b4-c7a41d12b218" (UID: "c862a7fb-546a-4955-87b4-c7a41d12b218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:18:13.229182 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.229161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps" (OuterVolumeSpecName: "kube-api-access-6fsps") pod "c862a7fb-546a-4955-87b4-c7a41d12b218" (UID: "c862a7fb-546a-4955-87b4-c7a41d12b218"). InnerVolumeSpecName "kube-api-access-6fsps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:18:13.327905 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.327871    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a7fb-546a-4955-87b4-c7a41d12b218-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.327905 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.327900    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fsps\" (UniqueName: \"kubernetes.io/projected/c862a7fb-546a-4955-87b4-c7a41d12b218-kube-api-access-6fsps\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.327905 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.327910    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.328376 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.327920    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c862a7fb-546a-4955-87b4-c7a41d12b218-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.979310 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.979272    2565 generic.go:356] "Generic (PLEG): container finished" podID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerID="595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd" exitCode=0
Jul 15 11:18:13.979491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.979317    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"
Jul 15 11:18:13.979491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.979355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" event={"ID":"c862a7fb-546a-4955-87b4-c7a41d12b218","Type":"ContainerDied","Data":"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"}
Jul 15 11:18:13.979491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.979408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f" event={"ID":"c862a7fb-546a-4955-87b4-c7a41d12b218","Type":"ContainerDied","Data":"5a0f43daa834414c93d5abea1535240494898fe46dafd4c001c716c0abe086ef"}
Jul 15 11:18:13.979491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.979432    2565 scope.go:117] "RemoveContainer" containerID="595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"
Jul 15 11:18:13.991489 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.991468    2565 scope.go:117] "RemoveContainer" containerID="a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0"
Jul 15 11:18:13.999986 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:13.999969    2565 scope.go:117] "RemoveContainer" containerID="595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"
Jul 15 11:18:14.000252 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:18:14.000234    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd\": container with ID starting with 595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd not found: ID does not exist" containerID="595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"
Jul 15 11:18:14.000306 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.000262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd"} err="failed to get container status \"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd\": rpc error: code = NotFound desc = could not find container \"595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd\": container with ID starting with 595fe4c5ab978a73cf979ae4a1c214edb71e2d9dd26a68bf5cc3ecdeddb16efd not found: ID does not exist"
Jul 15 11:18:14.000306 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.000277    2565 scope.go:117] "RemoveContainer" containerID="a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0"
Jul 15 11:18:14.000562 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:18:14.000523    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0\": container with ID starting with a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0 not found: ID does not exist" containerID="a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0"
Jul 15 11:18:14.000562 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.000572    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0"} err="failed to get container status \"a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0\": rpc error: code = NotFound desc = could not find container \"a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0\": container with ID starting with a6173fbc4cc90e2862b01507666a1c6e4307b7c43ba4b0bde92f5932e033efc0 not found: ID does not exist"
Jul 15 11:18:14.003416 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.003390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:18:14.008844 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.008824    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f"]
Jul 15 11:18:14.675219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:14.675186    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" path="/var/lib/kubelet/pods/c862a7fb-546a-4955-87b4-c7a41d12b218/volumes"
Jul 15 11:18:18.283989 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:18.283951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:18.389804 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:18.389775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:18.490546 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:18.490518    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:20.837182 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:20.837147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:20.903770 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:20.903737    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:21.347660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.347629    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"]
Jul 15 11:18:21.348065 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.348029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main"
Jul 15 11:18:21.348065 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.348056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main"
Jul 15 11:18:21.348164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.348073    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="storage-initializer"
Jul 15 11:18:21.348164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.348079    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="storage-initializer"
Jul 15 11:18:21.348164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.348132    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c862a7fb-546a-4955-87b4-c7a41d12b218" containerName="main"
Jul 15 11:18:21.352566 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.352545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:21.359472 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.359447    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"]
Jul 15 11:18:21.398437 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.398408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6hn\" (UniqueName: \"kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:21.464668 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.464635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:21.498868 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.498820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6hn\" (UniqueName: \"kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:21.508660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:21.508636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6hn\" (UniqueName: \"kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.018837 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.018800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.028689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.028662    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.119258 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.119221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.119431 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.119352    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.122189 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.122169    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") DevicePath \"csi-b8bbd7d42f53248572fa1a17fbcbd1d183f9079f87108ff68052984af4714998\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.338444 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.338414    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4767870384ba5a33f8c589260e567a3f76dd02f928fb126f91966e97c3d5e812/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.352425 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.352393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"e2e-pvc-model-download-qm8ll\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.368453 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.368432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:24.506035 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:24.506010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"]
Jul 15 11:18:24.507642 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:18:24.507617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1313084_7836_49b9_a6e0_a7ccb413d04a.slice/crio-a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570 WatchSource:0}: Error finding container a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570: Status 404 returned error can't find the container with id a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570
Jul 15 11:18:25.046816 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:25.046776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll" event={"ID":"a1313084-7836-49b9-a6e0-a7ccb413d04a","Type":"ContainerStarted","Data":"05d5a5d0e14b8f1a9c60c0407a534c7e21c8987ba9047d27358f55058beaad61"}
Jul 15 11:18:25.047213 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:25.046827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll" event={"ID":"a1313084-7836-49b9-a6e0-a7ccb413d04a","Type":"ContainerStarted","Data":"a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570"}
Jul 15 11:18:25.064810 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:25.064726    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll" podStartSLOduration=7.064703033 podStartE2EDuration="7.064703033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:18:25.063534228 +0000 UTC m=+1970.898278679" watchObservedRunningTime="2026-07-15 11:18:25.064703033 +0000 UTC m=+1970.899447484"
Jul 15 11:18:28.898160 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:28.898126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:18:31.077709 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:31.077679    2565 generic.go:356] "Generic (PLEG): container finished" podID="a1313084-7836-49b9-a6e0-a7ccb413d04a" containerID="05d5a5d0e14b8f1a9c60c0407a534c7e21c8987ba9047d27358f55058beaad61" exitCode=0
Jul 15 11:18:31.078091 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:31.077759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll" event={"ID":"a1313084-7836-49b9-a6e0-a7ccb413d04a","Type":"ContainerDied","Data":"05d5a5d0e14b8f1a9c60c0407a534c7e21c8987ba9047d27358f55058beaad61"}
Jul 15 11:18:32.229077 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.229028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:32.294000 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.293969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d6hn\" (UniqueName: \"kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn\") pod \"a1313084-7836-49b9-a6e0-a7ccb413d04a\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") "
Jul 15 11:18:32.294178 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.294109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"a1313084-7836-49b9-a6e0-a7ccb413d04a\" (UID: \"a1313084-7836-49b9-a6e0-a7ccb413d04a\") "
Jul 15 11:18:32.296502 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.296475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn" (OuterVolumeSpecName: "kube-api-access-8d6hn") pod "a1313084-7836-49b9-a6e0-a7ccb413d04a" (UID: "a1313084-7836-49b9-a6e0-a7ccb413d04a"). InnerVolumeSpecName "kube-api-access-8d6hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:18:32.298790 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.298769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "model-storage") pod "a1313084-7836-49b9-a6e0-a7ccb413d04a" (UID: "a1313084-7836-49b9-a6e0-a7ccb413d04a"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:18:32.395815 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.395749    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6hn\" (UniqueName: \"kubernetes.io/projected/a1313084-7836-49b9-a6e0-a7ccb413d04a-kube-api-access-8d6hn\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:18:32.395815 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.395790    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-137-92.ec2.internal\" "
Jul 15 11:18:32.506803 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.506772    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4") on node "ip-10-0-137-92.ec2.internal"
Jul 15 11:18:32.598222 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:32.598188    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"csi-b8bbd7d42f53248572fa1a17fbcbd1d183f9079f87108ff68052984af4714998\""
Jul 15 11:18:33.089320 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:33.089292    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll"
Jul 15 11:18:33.089320 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:33.089303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll" event={"ID":"a1313084-7836-49b9-a6e0-a7ccb413d04a","Type":"ContainerDied","Data":"a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570"}
Jul 15 11:18:33.089517 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:33.089342    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a382420bab3af4e90fd6857d69a57dbfc56d33c576c630b9d9bd41cd5f802570"
Jul 15 11:18:44.982282 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:44.982247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:45.082300 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:45.082268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:51.664771 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:51.664723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:51.958771 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:51.958692    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:18:59.262120 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:18:59.262082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:19:01.307510 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:19:01.307478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:20:34.766878 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:20:34.766850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:20:34.770886 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:20:34.770862    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:20:34.775519 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:20:34.775500    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:20:34.779251 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:20:34.779234    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:21:00.897391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:00.897358    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:21:00.903724 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:00.903694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:00.991872 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:00.991835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:08.898329 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:08.898292    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:21:15.918500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:15.918455    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="main" containerID="cri-o://2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7" gracePeriod=45
Jul 15 11:21:16.177006 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.176944    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:21:16.296953 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.296920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs\") pod \"8be114d7-1447-4eea-a823-029e5483452d\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") "
Jul 15 11:21:16.297174 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.296972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5cjx\" (UniqueName: \"kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx\") pod \"8be114d7-1447-4eea-a823-029e5483452d\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") "
Jul 15 11:21:16.297174 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.297010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds\") pod \"8be114d7-1447-4eea-a823-029e5483452d\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") "
Jul 15 11:21:16.297174 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.297095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location\") pod \"8be114d7-1447-4eea-a823-029e5483452d\" (UID: \"8be114d7-1447-4eea-a823-029e5483452d\") "
Jul 15 11:21:16.297355 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.297329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8be114d7-1447-4eea-a823-029e5483452d" (UID: "8be114d7-1447-4eea-a823-029e5483452d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:16.297819 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.297794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8be114d7-1447-4eea-a823-029e5483452d" (UID: "8be114d7-1447-4eea-a823-029e5483452d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:16.299296 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.299273    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx" (OuterVolumeSpecName: "kube-api-access-h5cjx") pod "8be114d7-1447-4eea-a823-029e5483452d" (UID: "8be114d7-1447-4eea-a823-029e5483452d"). InnerVolumeSpecName "kube-api-access-h5cjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:21:16.299296 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.299277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8be114d7-1447-4eea-a823-029e5483452d" (UID: "8be114d7-1447-4eea-a823-029e5483452d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:21:16.397645 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.397618    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be114d7-1447-4eea-a823-029e5483452d-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:21:16.397645 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.397641    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cjx\" (UniqueName: \"kubernetes.io/projected/8be114d7-1447-4eea-a823-029e5483452d-kube-api-access-h5cjx\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:21:16.397786 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.397650    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:21:16.397786 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.397659    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be114d7-1447-4eea-a823-029e5483452d-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:21:16.963317 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.963228    2565 generic.go:356] "Generic (PLEG): container finished" podID="8be114d7-1447-4eea-a823-029e5483452d" containerID="2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7" exitCode=0
Jul 15 11:21:16.963317 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.963284    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"
Jul 15 11:21:16.963317 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.963308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerDied","Data":"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"}
Jul 15 11:21:16.963813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.963356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6" event={"ID":"8be114d7-1447-4eea-a823-029e5483452d","Type":"ContainerDied","Data":"333fe309d63c9a00ea0c0d56ea9b5a29c9128c4bb285363aa8b43d87329297e1"}
Jul 15 11:21:16.963813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.963380    2565 scope.go:117] "RemoveContainer" containerID="2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"
Jul 15 11:21:16.974735 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.974707    2565 scope.go:117] "RemoveContainer" containerID="f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"
Jul 15 11:21:16.984717 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.984686    2565 scope.go:117] "RemoveContainer" containerID="2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"
Jul 15 11:21:16.985104 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:21:16.985080    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7\": container with ID starting with 2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7 not found: ID does not exist" containerID="2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"
Jul 15 11:21:16.985193 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.985113    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7"} err="failed to get container status \"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7\": rpc error: code = NotFound desc = could not find container \"2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7\": container with ID starting with 2e58d3bf0b8ef41d2194f453c3de1bd8bf3e14dfc8905417c92ce75bf16cb0f7 not found: ID does not exist"
Jul 15 11:21:16.985193 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.985131    2565 scope.go:117] "RemoveContainer" containerID="f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"
Jul 15 11:21:16.985338 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.985318    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:21:16.985404 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:21:16.985389    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07\": container with ID starting with f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07 not found: ID does not exist" containerID="f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"
Jul 15 11:21:16.985443 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.985409    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07"} err="failed to get container status \"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07\": rpc error: code = NotFound desc = could not find container \"f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07\": container with ID starting with f0a3148ca8997badeb70bacde3dbc5efe4fede4470d4677623e2f21c92756b07 not found: ID does not exist"
Jul 15 11:21:16.991525 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:16.991500    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6"]
Jul 15 11:21:18.674294 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:18.674264    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be114d7-1447-4eea-a823-029e5483452d" path="/var/lib/kubelet/pods/8be114d7-1447-4eea-a823-029e5483452d/volumes"
Jul 15 11:21:24.212014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:24.211961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:30.019936 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.019899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:30.270710 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.270632    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:21:30.271016 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271002    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="main"
Jul 15 11:21:30.271089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="main"
Jul 15 11:21:30.271089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="storage-initializer"
Jul 15 11:21:30.271089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="storage-initializer"
Jul 15 11:21:30.271089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271055    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1313084-7836-49b9-a6e0-a7ccb413d04a" containerName="storage-initializer"
Jul 15 11:21:30.271089 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271061    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1313084-7836-49b9-a6e0-a7ccb413d04a" containerName="storage-initializer"
Jul 15 11:21:30.271248 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271121    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be114d7-1447-4eea-a823-029e5483452d" containerName="main"
Jul 15 11:21:30.271248 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.271133    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1313084-7836-49b9-a6e0-a7ccb413d04a" containerName="storage-initializer"
Jul 15 11:21:30.275866 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.275844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.278633 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.278612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:21:30.278754 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.278686    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:21:30.278754 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.278705    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:21:30.278754 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.278723    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 15 11:21:30.278916 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.278686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-94ww4\""
Jul 15 11:21:30.279507 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.279487    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:21:30.286228 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.286205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:21:30.304254 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.304216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:30.420412 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.420368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.420594 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.420422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.420594 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.420475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.420594 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.420580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv27f\" (UniqueName: \"kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.521950 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.521859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.521950 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.521947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv27f\" (UniqueName: \"kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.522191 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.521982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.522191 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.522011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.522491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.522458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.522491 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.522474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.524581 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.524560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.530811 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.530785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv27f\" (UniqueName: \"kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.586222 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.586189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:30.736753 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:30.736723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:21:30.737998 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:21:30.737969    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfb128b_bff4_451d_a6ac_9e37caceafe1.slice/crio-6c23d6f8aed6090a61919b40d324476979b91464893966ceb3ce4cda0b48d503 WatchSource:0}: Error finding container 6c23d6f8aed6090a61919b40d324476979b91464893966ceb3ce4cda0b48d503: Status 404 returned error can't find the container with id 6c23d6f8aed6090a61919b40d324476979b91464893966ceb3ce4cda0b48d503
Jul 15 11:21:31.043755 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:31.043672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerStarted","Data":"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"}
Jul 15 11:21:31.043755 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:31.043726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerStarted","Data":"6c23d6f8aed6090a61919b40d324476979b91464893966ceb3ce4cda0b48d503"}
Jul 15 11:21:32.051139 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:32.051103    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerID="2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd" exitCode=0
Jul 15 11:21:32.051498 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:32.051158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerDied","Data":"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"}
Jul 15 11:21:32.141776 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:32.141740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:33.057596 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:33.057555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerStarted","Data":"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"}
Jul 15 11:21:33.058001 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:33.057706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:21:33.078583 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:33.078535    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" podStartSLOduration=2.4085129 podStartE2EDuration="3.078521684s" totalImagesPullingTime="0s" totalInitContainerRuntime="670.008784ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:21:33.077001626 +0000 UTC m=+2158.911746106" watchObservedRunningTime="2026-07-15 11:21:33.078521684 +0000 UTC m=+2158.913266133"
Jul 15 11:21:51.786018 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:51.785976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:53.690485 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:53.690445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:21:53.772006 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:21:53.771971    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:22:04.065146 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:22:04.065114    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:22:24.313721 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:22:24.313685    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:22:24.665569 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:22:24.665527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:22:33.013313 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:22:33.013272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:25:25.101654 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:25.101619    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:25:34.802792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:34.802762    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:25:34.807030 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:34.807006    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:25:34.812420 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:34.812403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:25:34.816277 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:34.816261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:25:44.780436 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:44.780407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:25:44.854052 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:44.854002    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:25:59.174275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:59.174192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:25:59.189654 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:59.189620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:25:59.263237 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:25:59.263200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:04.064109 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:04.064072    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:26:14.064571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.064527    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:26:14.211227 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.211188    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main" containerID="cri-o://785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227" gracePeriod=45
Jul 15 11:26:14.475498 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.475474    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:26:14.563991 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.563954    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerID="785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227" exitCode=0
Jul 15 11:26:14.564164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.564009    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"
Jul 15 11:26:14.564164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.564057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerDied","Data":"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"}
Jul 15 11:26:14.564164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.564140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4" event={"ID":"ebfb128b-bff4-451d-a6ac-9e37caceafe1","Type":"ContainerDied","Data":"6c23d6f8aed6090a61919b40d324476979b91464893966ceb3ce4cda0b48d503"}
Jul 15 11:26:14.564164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.564162    2565 scope.go:117] "RemoveContainer" containerID="785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"
Jul 15 11:26:14.569537 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.569517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv27f\" (UniqueName: \"kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f\") pod \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") "
Jul 15 11:26:14.569674 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.569571    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs\") pod \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") "
Jul 15 11:26:14.569674 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.569626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location\") pod \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") "
Jul 15 11:26:14.569674 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.569647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds\") pod \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\" (UID: \"ebfb128b-bff4-451d-a6ac-9e37caceafe1\") "
Jul 15 11:26:14.569972 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.569943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ebfb128b-bff4-451d-a6ac-9e37caceafe1" (UID: "ebfb128b-bff4-451d-a6ac-9e37caceafe1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:14.570369 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.570348    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebfb128b-bff4-451d-a6ac-9e37caceafe1" (UID: "ebfb128b-bff4-451d-a6ac-9e37caceafe1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:14.571733 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.571711    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebfb128b-bff4-451d-a6ac-9e37caceafe1" (UID: "ebfb128b-bff4-451d-a6ac-9e37caceafe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:14.571840 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.571782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f" (OuterVolumeSpecName: "kube-api-access-rv27f") pod "ebfb128b-bff4-451d-a6ac-9e37caceafe1" (UID: "ebfb128b-bff4-451d-a6ac-9e37caceafe1"). InnerVolumeSpecName "kube-api-access-rv27f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:14.573923 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.573897    2565 scope.go:117] "RemoveContainer" containerID="2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"
Jul 15 11:26:14.591622 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.591601    2565 scope.go:117] "RemoveContainer" containerID="785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"
Jul 15 11:26:14.592035 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:26:14.592009    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227\": container with ID starting with 785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227 not found: ID does not exist" containerID="785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"
Jul 15 11:26:14.592168 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.592058    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227"} err="failed to get container status \"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227\": rpc error: code = NotFound desc = could not find container \"785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227\": container with ID starting with 785d8a47cdcdbbc7de33e7c93a22484826bdb4b3f5dac15932338ebc79e72227 not found: ID does not exist"
Jul 15 11:26:14.592168 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.592083    2565 scope.go:117] "RemoveContainer" containerID="2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"
Jul 15 11:26:14.592386 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:26:14.592366    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd\": container with ID starting with 2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd not found: ID does not exist" containerID="2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"
Jul 15 11:26:14.592426 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.592393    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd"} err="failed to get container status \"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd\": rpc error: code = NotFound desc = could not find container \"2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd\": container with ID starting with 2e54d0bc4465dfbf8cd68af13c30693cca3c48fa2ad00c5fb00446a6d9f55ccd not found: ID does not exist"
Jul 15 11:26:14.671148 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.671116    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:14.671148 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.671146    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:14.671334 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.671163    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv27f\" (UniqueName: \"kubernetes.io/projected/ebfb128b-bff4-451d-a6ac-9e37caceafe1-kube-api-access-rv27f\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:14.671334 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.671176    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfb128b-bff4-451d-a6ac-9e37caceafe1-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:14.882764 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.882685    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:26:14.888401 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:14.888378    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4"]
Jul 15 11:26:16.674246 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:16.674213    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" path="/var/lib/kubelet/pods/ebfb128b-bff4-451d-a6ac-9e37caceafe1/volumes"
Jul 15 11:26:17.969988 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:17.969953    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:18.064753 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:18.064716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:18.165745 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:18.165713    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:18.316402 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:18.316318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:20.199529 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:20.199953 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199883    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main"
Jul 15 11:26:20.199953 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199898    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main"
Jul 15 11:26:20.199953 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199919    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="storage-initializer"
Jul 15 11:26:20.199953 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199926    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="storage-initializer"
Jul 15 11:26:20.200115 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.199984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebfb128b-bff4-451d-a6ac-9e37caceafe1" containerName="main"
Jul 15 11:26:20.204574 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.204553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.207078 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.207055    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 11:26:20.207423 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.207084    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 11:26:20.207423 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.207407    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 11:26:20.208319 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.208125    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-mm46t\""
Jul 15 11:26:20.226462 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.226431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:20.252675 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.252639    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:20.313922 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.313864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.313922 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.313903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.313922 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.313922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.313996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.314086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.314121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfk22\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.314149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314164 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.314164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.314348 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.314196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.331664 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.331596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:20.415638 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfk22\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.415796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416162 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416162 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.415877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416162 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.416102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416276 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.416209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416374 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.416357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416429 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.416382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.416684 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.416623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.418394 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.418374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.418484 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.418427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.424932 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.424913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfk22\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.425023 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.424960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token\") pod \"router-gateway-2-openshift-default-5b6bc58868-s5jth\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.516256 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.516159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:20.665115 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.665088    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:20.667332 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:26:20.667305    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb100fc47_ae29_426d_8a15_8ab96a147d9c.slice/crio-2da7ebfe0d278e505b05819eaaabeeaf765d1e2bee42072a4d59da8e20c940ab WatchSource:0}: Error finding container 2da7ebfe0d278e505b05819eaaabeeaf765d1e2bee42072a4d59da8e20c940ab: Status 404 returned error can't find the container with id 2da7ebfe0d278e505b05819eaaabeeaf765d1e2bee42072a4d59da8e20c940ab
Jul 15 11:26:20.670072 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.670019    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:20.670202 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.670108    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:20.670202 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:20.670139    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:21.606002 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:21.605964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" event={"ID":"b100fc47-ae29-426d-8a15-8ab96a147d9c","Type":"ContainerStarted","Data":"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"}
Jul 15 11:26:21.606408 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:21.606022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" event={"ID":"b100fc47-ae29-426d-8a15-8ab96a147d9c","Type":"ContainerStarted","Data":"2da7ebfe0d278e505b05819eaaabeeaf765d1e2bee42072a4d59da8e20c940ab"}
Jul 15 11:26:21.632470 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:21.632419    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" podStartSLOduration=1.632402977 podStartE2EDuration="1.632402977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:21.628931048 +0000 UTC m=+2447.463675498" watchObservedRunningTime="2026-07-15 11:26:21.632402977 +0000 UTC m=+2447.467147427"
Jul 15 11:26:22.516540 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:22.516508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:22.521675 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:22.521646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:22.611930 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:22.611901    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:22.612821 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:22.612802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:23.278887 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.278853    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"]
Jul 15 11:26:23.282282 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.282258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.299203 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.299176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"]
Jul 15 11:26:23.344640 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxf4j\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-kube-api-access-jxf4j\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aacc01e-5225-4093-8788-5cd9b555b523-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.344835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.345099 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.345099 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.344883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aacc01e-5225-4093-8788-5cd9b555b523-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445786 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxf4j\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-kube-api-access-jxf4j\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445974 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445974 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445974 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445974 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.445974 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.445932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aacc01e-5225-4093-8788-5cd9b555b523-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446288 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446288 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446288 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aacc01e-5225-4093-8788-5cd9b555b523-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446446 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-certs\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446446 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-workload-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446446 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-credential-socket\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446624 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-data\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.446748 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.446727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aacc01e-5225-4093-8788-5cd9b555b523-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.448670 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.448649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aacc01e-5225-4093-8788-5cd9b555b523-istio-envoy\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.448793 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.448747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aacc01e-5225-4093-8788-5cd9b555b523-istio-podinfo\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.454152 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.454126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-istio-token\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.454152 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.454145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxf4j\" (UniqueName: \"kubernetes.io/projected/9aacc01e-5225-4093-8788-5cd9b555b523-kube-api-access-jxf4j\") pod \"router-gateway-2-openshift-default-65d6b86cbf-n2jmt\" (UID: \"9aacc01e-5225-4093-8788-5cd9b555b523\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.593256 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.593219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:23.740637 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.740599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"]
Jul 15 11:26:23.743521 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:26:23.743491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aacc01e_5225_4093_8788_5cd9b555b523.slice/crio-16046c0037cbaae84cf69a21632b7ed5df23d7e8a152d4f005d72ad250ae34ec WatchSource:0}: Error finding container 16046c0037cbaae84cf69a21632b7ed5df23d7e8a152d4f005d72ad250ae34ec: Status 404 returned error can't find the container with id 16046c0037cbaae84cf69a21632b7ed5df23d7e8a152d4f005d72ad250ae34ec
Jul 15 11:26:23.745742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.745706    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:23.745809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.745775    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:23.745809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:23.745803    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:24.168601 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:24.168561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:24.624683 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:24.624639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" event={"ID":"9aacc01e-5225-4093-8788-5cd9b555b523","Type":"ContainerStarted","Data":"0355ae9be04d7a563dc71232b84b4fefa714457df438bc9b885e6eb918db97a0"}
Jul 15 11:26:24.624872 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:24.624725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" event={"ID":"9aacc01e-5225-4093-8788-5cd9b555b523","Type":"ContainerStarted","Data":"16046c0037cbaae84cf69a21632b7ed5df23d7e8a152d4f005d72ad250ae34ec"}
Jul 15 11:26:24.650910 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:24.650856    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podStartSLOduration=1.6508157140000002 podStartE2EDuration="1.650815714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:24.649655537 +0000 UTC m=+2450.484400014" watchObservedRunningTime="2026-07-15 11:26:24.650815714 +0000 UTC m=+2450.485560164"
Jul 15 11:26:25.316656 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.316619    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:25.593976 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.593941    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:25.599001 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.598977    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:25.629786 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.629757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:25.630797 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.630774    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:25.689983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.689956    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:25.690197 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:25.690175    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" podUID="b100fc47-ae29-426d-8a15-8ab96a147d9c" containerName="istio-proxy" containerID="cri-o://f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6" gracePeriod=30
Jul 15 11:26:30.982538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:30.982512    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:31.126938 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.126902    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.126938 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.126938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.126980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfk22\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127224    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket\") pod \"b100fc47-ae29-426d-8a15-8ab96a147d9c\" (UID: \"b100fc47-ae29-426d-8a15-8ab96a147d9c\") "
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127377    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:31.127492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:26:31.127792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:31.127792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127588    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.127792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127607    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-data\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.127792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127621    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-workload-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.127792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.127633    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b100fc47-ae29-426d-8a15-8ab96a147d9c-istiod-ca-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.129427 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.129401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:31.129427 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.129418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22" (OuterVolumeSpecName: "kube-api-access-lfk22") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "kube-api-access-lfk22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:31.129803 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.129781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:31.129846 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.129790    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b100fc47-ae29-426d-8a15-8ab96a147d9c" (UID: "b100fc47-ae29-426d-8a15-8ab96a147d9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 11:26:31.228389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.228359    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfk22\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-kube-api-access-lfk22\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.228389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.228387    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-podinfo\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.228389 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.228396    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-credential-socket\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.228626 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.228407    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.228626 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.228416    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b100fc47-ae29-426d-8a15-8ab96a147d9c-istio-envoy\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:26:31.672783 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.672743    2565 generic.go:356] "Generic (PLEG): container finished" podID="b100fc47-ae29-426d-8a15-8ab96a147d9c" containerID="f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6" exitCode=0
Jul 15 11:26:31.672965 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.672823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" event={"ID":"b100fc47-ae29-426d-8a15-8ab96a147d9c","Type":"ContainerDied","Data":"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"}
Jul 15 11:26:31.672965 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.672882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth" event={"ID":"b100fc47-ae29-426d-8a15-8ab96a147d9c","Type":"ContainerDied","Data":"2da7ebfe0d278e505b05819eaaabeeaf765d1e2bee42072a4d59da8e20c940ab"}
Jul 15 11:26:31.672965 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.672913    2565 scope.go:117] "RemoveContainer" containerID="f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"
Jul 15 11:26:31.673214 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.673083    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"
Jul 15 11:26:31.687208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.687164    2565 scope.go:117] "RemoveContainer" containerID="f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"
Jul 15 11:26:31.687645 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:26:31.687615    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6\": container with ID starting with f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6 not found: ID does not exist" containerID="f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"
Jul 15 11:26:31.687721 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.687653    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6"} err="failed to get container status \"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6\": rpc error: code = NotFound desc = could not find container \"f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6\": container with ID starting with f4715088daf2f9052db87ec9a518e5b764106657dbf2ac2ebce65f9e05c0c1d6 not found: ID does not exist"
Jul 15 11:26:31.700902 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.700879    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:31.707215 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:31.707193    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth"]
Jul 15 11:26:32.673733 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:32.673700    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b100fc47-ae29-426d-8a15-8ab96a147d9c" path="/var/lib/kubelet/pods/b100fc47-ae29-426d-8a15-8ab96a147d9c/volumes"
Jul 15 11:26:34.616351 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:34.616316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:35.106166 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.106129    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:26:35.107257 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.106552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b100fc47-ae29-426d-8a15-8ab96a147d9c" containerName="istio-proxy"
Jul 15 11:26:35.107257 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.106565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b100fc47-ae29-426d-8a15-8ab96a147d9c" containerName="istio-proxy"
Jul 15 11:26:35.107257 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.106627    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b100fc47-ae29-426d-8a15-8ab96a147d9c" containerName="istio-proxy"
Jul 15 11:26:35.108765 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.108748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.111420 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.111383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:26:35.111552 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.111474    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:26:35.111968 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.111949    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-xnnfk\""
Jul 15 11:26:35.112326 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.112310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 15 11:26:35.125016 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.124988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:26:35.222200 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.222117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:35.267421 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.267388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.267590 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.267432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.267590 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.267460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hd2\" (UniqueName: \"kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.267590 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.267549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368396 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hd2\" (UniqueName: \"kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368818 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.368870 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.368798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.371172 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.371142    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.372002 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.371977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:26:35.377742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.377706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hd2\" (UniqueName: \"kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2\") pod \"router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.418692 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.418664    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:35.558481 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.558455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:26:35.559626 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:26:35.559601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4395ab_a7a8_41d9_aa29_013e9530af2c.slice/crio-cb48da661dcb75983ae2d79b800ccc700ddc17219483408ea375bf0455341020 WatchSource:0}: Error finding container cb48da661dcb75983ae2d79b800ccc700ddc17219483408ea375bf0455341020: Status 404 returned error can't find the container with id cb48da661dcb75983ae2d79b800ccc700ddc17219483408ea375bf0455341020
Jul 15 11:26:35.697712 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.697676    2565 generic.go:356] "Generic (PLEG): container finished" podID="9aacc01e-5225-4093-8788-5cd9b555b523" containerID="0355ae9be04d7a563dc71232b84b4fefa714457df438bc9b885e6eb918db97a0" exitCode=0
Jul 15 11:26:35.698153 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.697758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" event={"ID":"9aacc01e-5225-4093-8788-5cd9b555b523","Type":"ContainerDied","Data":"0355ae9be04d7a563dc71232b84b4fefa714457df438bc9b885e6eb918db97a0"}
Jul 15 11:26:35.698225 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.698200    2565 scope.go:117] "RemoveContainer" containerID="0355ae9be04d7a563dc71232b84b4fefa714457df438bc9b885e6eb918db97a0"
Jul 15 11:26:35.699307 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.699279    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:35.699425 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.699364    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:35.699425 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.699410    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 15 11:26:35.699515 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.699457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerStarted","Data":"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"}
Jul 15 11:26:35.699515 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:35.699501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerStarted","Data":"cb48da661dcb75983ae2d79b800ccc700ddc17219483408ea375bf0455341020"}
Jul 15 11:26:36.706529 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:36.706442    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerID="b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4" exitCode=0
Jul 15 11:26:36.706529 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:36.706519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerDied","Data":"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"}
Jul 15 11:26:36.708638 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:36.708619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" event={"ID":"9aacc01e-5225-4093-8788-5cd9b555b523","Type":"ContainerStarted","Data":"125b13075f9e90d3eccc04bb7c8b7bfa25084583ced4ef3a1308cf75a1098695"}
Jul 15 11:26:37.593466 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:37.593431    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:37.594681 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:37.594650    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podUID="9aacc01e-5225-4093-8788-5cd9b555b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 15 11:26:37.715683 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:37.715642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerStarted","Data":"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"}
Jul 15 11:26:37.716082 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:37.715846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:26:37.747358 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:37.747305    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" podStartSLOduration=2.04128899 podStartE2EDuration="2.74728557s" totalImagesPullingTime="0s" totalInitContainerRuntime="705.99658ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:37.745339663 +0000 UTC m=+2463.580084113" watchObservedRunningTime="2026-07-15 11:26:37.74728557 +0000 UTC m=+2463.582030023"
Jul 15 11:26:38.593685 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:38.593636    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podUID="9aacc01e-5225-4093-8788-5cd9b555b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 15 11:26:39.593709 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:39.593658    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podUID="9aacc01e-5225-4093-8788-5cd9b555b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 15 11:26:40.594468 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:40.594423    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podUID="9aacc01e-5225-4093-8788-5cd9b555b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 15 11:26:40.630709 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:40.630675    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:41.594229 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:41.594186    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt" podUID="9aacc01e-5225-4093-8788-5cd9b555b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 15 11:26:42.598364 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:42.598336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:42.599339 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:42.599323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt"
Jul 15 11:26:46.519680 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:26:46.519642    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:27:06.334309 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:27:06.334267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:27:08.722323 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:27:08.722293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:27:29.788860 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:27:29.788773    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:27:41.337539 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:27:41.337501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:27:51.313140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:27:51.313104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:28:43.751361 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:28:43.751323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:28:43.851849 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:28:43.851820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:30:14.298268 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:14.298236    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:30:14.331094 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:14.331037    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:30:15.961699 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:15.961658    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:30:18.722776 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:18.722740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:30:28.722503 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:28.722412    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:30:29.319290 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.319248    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main" containerID="cri-o://0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291" gracePeriod=45
Jul 15 11:30:29.583234 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.583211    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:30:29.606240 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.606216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs\") pod \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") "
Jul 15 11:30:29.606364 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.606251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7hd2\" (UniqueName: \"kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2\") pod \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") "
Jul 15 11:30:29.606364 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.606281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds\") pod \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") "
Jul 15 11:30:29.606364 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.606358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location\") pod \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\" (UID: \"9c4395ab-a7a8-41d9-aa29-013e9530af2c\") "
Jul 15 11:30:29.606620 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.606589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9c4395ab-a7a8-41d9-aa29-013e9530af2c" (UID: "9c4395ab-a7a8-41d9-aa29-013e9530af2c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:29.607242 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.607190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c4395ab-a7a8-41d9-aa29-013e9530af2c" (UID: "9c4395ab-a7a8-41d9-aa29-013e9530af2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:29.608886 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.608863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c4395ab-a7a8-41d9-aa29-013e9530af2c" (UID: "9c4395ab-a7a8-41d9-aa29-013e9530af2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:30:29.609128 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.609098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2" (OuterVolumeSpecName: "kube-api-access-z7hd2") pod "9c4395ab-a7a8-41d9-aa29-013e9530af2c" (UID: "9c4395ab-a7a8-41d9-aa29-013e9530af2c"). InnerVolumeSpecName "kube-api-access-z7hd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:30:29.707813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.707778    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:30:29.707813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.707810    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hd2\" (UniqueName: \"kubernetes.io/projected/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kube-api-access-z7hd2\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:30:29.707813 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.707820    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:30:29.708096 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:29.707830    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c4395ab-a7a8-41d9-aa29-013e9530af2c-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.001482 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.001389    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerID="0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291" exitCode=0
Jul 15 11:30:30.001482 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.001451    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"
Jul 15 11:30:30.001928 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.001476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerDied","Data":"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"}
Jul 15 11:30:30.001928 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.001554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d" event={"ID":"9c4395ab-a7a8-41d9-aa29-013e9530af2c","Type":"ContainerDied","Data":"cb48da661dcb75983ae2d79b800ccc700ddc17219483408ea375bf0455341020"}
Jul 15 11:30:30.001928 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.001575    2565 scope.go:117] "RemoveContainer" containerID="0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"
Jul 15 11:30:30.011426 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.011408    2565 scope.go:117] "RemoveContainer" containerID="b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"
Jul 15 11:30:30.019713 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.019698    2565 scope.go:117] "RemoveContainer" containerID="0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"
Jul 15 11:30:30.019963 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:30:30.019943    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291\": container with ID starting with 0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291 not found: ID does not exist" containerID="0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"
Jul 15 11:30:30.020034 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.019970    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291"} err="failed to get container status \"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291\": rpc error: code = NotFound desc = could not find container \"0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291\": container with ID starting with 0463df94b81595b278b8f617bc5ca64bdaa1256b9d5c722493dba692bceee291 not found: ID does not exist"
Jul 15 11:30:30.020034 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.019988    2565 scope.go:117] "RemoveContainer" containerID="b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"
Jul 15 11:30:30.020255 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:30:30.020238    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4\": container with ID starting with b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4 not found: ID does not exist" containerID="b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"
Jul 15 11:30:30.020298 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.020260    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4"} err="failed to get container status \"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4\": rpc error: code = NotFound desc = could not find container \"b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4\": container with ID starting with b7cf726e45ac95b9244705db71d00379f8a0c5b67cbd4482c54409620e4e6fc4 not found: ID does not exist"
Jul 15 11:30:30.025264 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.025238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:30:30.029679 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.029649    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d"]
Jul 15 11:30:30.675255 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:30.675221    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" path="/var/lib/kubelet/pods/9c4395ab-a7a8-41d9-aa29-013e9530af2c/volumes"
Jul 15 11:30:33.558247 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:33.558213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:30:34.838905 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:34.838866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:30:34.843278 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:34.843258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:30:34.848622 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:34.848605    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:30:34.852848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:34.852830    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:30:40.385910 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:30:40.385877    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:32:45.715113 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:32:45.715077    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:18.514630 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.514589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:18.583676 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.583636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:18.614475 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614438    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:33:18.614849 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614836    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="storage-initializer"
Jul 15 11:33:18.614893 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614850    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="storage-initializer"
Jul 15 11:33:18.614893 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614865    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main"
Jul 15 11:33:18.614893 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614871    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main"
Jul 15 11:33:18.614989 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.614933    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4395ab-a7a8-41d9-aa29-013e9530af2c" containerName="main"
Jul 15 11:33:18.617538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.617522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.619973 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.619950    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:33:18.620551 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.620534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-l8jnh\""
Jul 15 11:33:18.620633 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.620558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:33:18.620633 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.620539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 15 11:33:18.629667 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.629639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:33:18.671817 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.671787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.671826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.671845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.671864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.671943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672014 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.672009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.672321 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.672030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmx6z\" (UniqueName: \"kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773611 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773611 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773857 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx6z\" (UniqueName: \"kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773857 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773969 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773969 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.773969 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.774176 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.773999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.774281 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.774257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.774368 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.774339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.774583 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.774561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.776161 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.776136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.776270 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.776208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.782085 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.782063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx6z\" (UniqueName: \"kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:18.928667 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:18.928627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:33:19.009488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.009434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:33:19.013612 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.013583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.019798 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.019763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-bjnc2\""
Jul 15 11:33:19.032492 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.032468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:33:19.076858 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.076822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xmw\" (UniqueName: \"kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.077070 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.076868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.077070 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.076938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.077172 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.077116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.088615 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.088563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:33:19.090510 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:33:19.090477    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e720bd5_2127_421f_8af4_8bc510d75aa1.slice/crio-f41e9e0c34382efa81dcacf11c14ef75acf7152f2c116448e10e9a633191a4ca WatchSource:0}: Error finding container f41e9e0c34382efa81dcacf11c14ef75acf7152f2c116448e10e9a633191a4ca: Status 404 returned error can't find the container with id f41e9e0c34382efa81dcacf11c14ef75acf7152f2c116448e10e9a633191a4ca
Jul 15 11:33:19.180071 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.179991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.180259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.180214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.180259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.180255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xmw\" (UniqueName: \"kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.180391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.180295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.180391 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.180330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.181000 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.180970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.182993 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.182971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.189484 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.189464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xmw\" (UniqueName: \"kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.327285 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.327255    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:19.477420 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.477226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:33:19.479577 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:33:19.479546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55e24b0_7b89_402b_a500_127cfdb798b5.slice/crio-8130804cf5e0a5c33ae0454257fe111c6f88d728e7fb886cd3c8e9f8979d01e8 WatchSource:0}: Error finding container 8130804cf5e0a5c33ae0454257fe111c6f88d728e7fb886cd3c8e9f8979d01e8: Status 404 returned error can't find the container with id 8130804cf5e0a5c33ae0454257fe111c6f88d728e7fb886cd3c8e9f8979d01e8
Jul 15 11:33:19.922153 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.922112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerStarted","Data":"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"}
Jul 15 11:33:19.922634 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.922179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerStarted","Data":"8130804cf5e0a5c33ae0454257fe111c6f88d728e7fb886cd3c8e9f8979d01e8"}
Jul 15 11:33:19.923902 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.923870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerStarted","Data":"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"}
Jul 15 11:33:19.924017 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:19.923926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerStarted","Data":"f41e9e0c34382efa81dcacf11c14ef75acf7152f2c116448e10e9a633191a4ca"}
Jul 15 11:33:20.337347 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:20.337312    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:20.931272 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:20.931236    2565 generic.go:356] "Generic (PLEG): container finished" podID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerID="6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac" exitCode=0
Jul 15 11:33:20.931743 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:20.931321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerDied","Data":"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"}
Jul 15 11:33:21.938915 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:21.938880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerStarted","Data":"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"}
Jul 15 11:33:21.939470 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:21.939084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:21.965623 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:21.965555    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" podStartSLOduration=3.242655004 podStartE2EDuration="3.965534248s" totalImagesPullingTime="0s" totalInitContainerRuntime="722.879244ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:33:21.964080767 +0000 UTC m=+2867.798825297" watchObservedRunningTime="2026-07-15 11:33:21.965534248 +0000 UTC m=+2867.800278700"
Jul 15 11:33:22.946278 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:22.946240    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerID="06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc" exitCode=0
Jul 15 11:33:22.946662 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:22.946316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerDied","Data":"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"}
Jul 15 11:33:22.947661 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:22.947645    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 11:33:51.148238 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:51.148196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerStarted","Data":"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"}
Jul 15 11:33:51.171399 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:51.171340    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.586451904 podStartE2EDuration="33.171323628s" totalImagesPullingTime="27.608629264s" totalInitContainerRuntime="2.97624246s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:33:51.169824248 +0000 UTC m=+2897.004568709" watchObservedRunningTime="2026-07-15 11:33:51.171323628 +0000 UTC m=+2897.006068077"
Jul 15 11:33:52.948316 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:52.948287    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:33:52.970773 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:52.970736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:55.625605 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:55.625555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:59.501196 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:59.501161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:33:59.563387 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:33:59.563343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:34:10.438590 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:34:10.438542    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:34:40.958363 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:34:40.958323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:34:41.382407 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:34:41.382360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:35:03.957938 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:35:03.957890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:35:34.877218 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:35:34.877192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:35:34.881075 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:35:34.881035    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:35:34.887710 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:35:34.887691    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:35:34.891361 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:35:34.891341    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:36:42.859015 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:42.858913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:43.303629 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:43.303543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:43.403095 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:43.403034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:53.736817 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.736784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:36:53.740737 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.740717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.743377 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.743349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 15 11:36:53.755107 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.755079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:36:53.765614 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.765581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:53.889445 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889659 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslxs\" (UniqueName: \"kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.889835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.889744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990538 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslxs\" (UniqueName: \"kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.990885 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.991073 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.990985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.991073 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.991023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.991190 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.991117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.991190 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.991134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.993067 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.993033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.993259 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.993237    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:53.999603 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:53.999581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslxs\" (UniqueName: \"kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs\") pod \"stop-feature-test-kserve-77469bd45-b6kqn\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:54.052684 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:54.052649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:36:54.196896 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:54.196860    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:54.209421 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:54.209394    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:36:54.210709 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:36:54.210669    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda627cd05_1287_4b57_94c6_205f35050486.slice/crio-a1f9650fcd6a7abe7179c090f8a3031d17c1af4ebbd837b99ea7c647f515ab7b WatchSource:0}: Error finding container a1f9650fcd6a7abe7179c090f8a3031d17c1af4ebbd837b99ea7c647f515ab7b: Status 404 returned error can't find the container with id a1f9650fcd6a7abe7179c090f8a3031d17c1af4ebbd837b99ea7c647f515ab7b
Jul 15 11:36:54.222768 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:54.222735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerStarted","Data":"a1f9650fcd6a7abe7179c090f8a3031d17c1af4ebbd837b99ea7c647f515ab7b"}
Jul 15 11:36:55.229473 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:55.229437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerStarted","Data":"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"}
Jul 15 11:36:57.244073 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:57.244021    2565 generic.go:356] "Generic (PLEG): container finished" podID="a627cd05-1287-4b57-94c6-205f35050486" containerID="6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612" exitCode=0
Jul 15 11:36:57.244533 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:57.244089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerDied","Data":"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"}
Jul 15 11:36:57.254418 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:57.254390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:36:57.267397 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:57.267231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:57.336338 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:57.336306    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:36:58.253536 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:58.253504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerStarted","Data":"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"}
Jul 15 11:36:58.277636 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:58.277580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podStartSLOduration=2.397186639 podStartE2EDuration="5.277563513s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.880376874s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:36:58.275679433 +0000 UTC m=+3084.110423922" watchObservedRunningTime="2026-07-15 11:36:58.277563513 +0000 UTC m=+3084.112307964"
Jul 15 11:36:59.053195 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:36:59.053133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:37:02.948274 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:02.948229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:37:04.053480 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:04.053433    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:37:04.055465 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:04.055432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:37:10.862024 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:10.861980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:37:12.284332 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.284283    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="main" containerID="cri-o://9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c" gracePeriod=45
Jul 15 11:37:12.551740 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.551711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:37:12.657561 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.657521    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location\") pod \"c55e24b0-7b89-402b-a500-127cfdb798b5\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") "
Jul 15 11:37:12.657719 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.657614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds\") pod \"c55e24b0-7b89-402b-a500-127cfdb798b5\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") "
Jul 15 11:37:12.657719 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.657689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs\") pod \"c55e24b0-7b89-402b-a500-127cfdb798b5\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") "
Jul 15 11:37:12.657853 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.657719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78xmw\" (UniqueName: \"kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw\") pod \"c55e24b0-7b89-402b-a500-127cfdb798b5\" (UID: \"c55e24b0-7b89-402b-a500-127cfdb798b5\") "
Jul 15 11:37:12.657909 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.657841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c55e24b0-7b89-402b-a500-127cfdb798b5" (UID: "c55e24b0-7b89-402b-a500-127cfdb798b5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:12.658059 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.658024    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-tokenizer-uds\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:12.658310 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.658281    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c55e24b0-7b89-402b-a500-127cfdb798b5" (UID: "c55e24b0-7b89-402b-a500-127cfdb798b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:12.660066 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.660011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c55e24b0-7b89-402b-a500-127cfdb798b5" (UID: "c55e24b0-7b89-402b-a500-127cfdb798b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:37:12.660507 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.660482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw" (OuterVolumeSpecName: "kube-api-access-78xmw") pod "c55e24b0-7b89-402b-a500-127cfdb798b5" (UID: "c55e24b0-7b89-402b-a500-127cfdb798b5"). InnerVolumeSpecName "kube-api-access-78xmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:37:12.758624 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.758589    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55e24b0-7b89-402b-a500-127cfdb798b5-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:12.758624 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.758617    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xmw\" (UniqueName: \"kubernetes.io/projected/c55e24b0-7b89-402b-a500-127cfdb798b5-kube-api-access-78xmw\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:12.758624 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:12.758626    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55e24b0-7b89-402b-a500-127cfdb798b5-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.351417 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.351374    2565 generic.go:356] "Generic (PLEG): container finished" podID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerID="9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c" exitCode=0
Jul 15 11:37:13.351838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.351440    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"
Jul 15 11:37:13.351838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.351457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerDied","Data":"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"}
Jul 15 11:37:13.351838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.351543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2" event={"ID":"c55e24b0-7b89-402b-a500-127cfdb798b5","Type":"ContainerDied","Data":"8130804cf5e0a5c33ae0454257fe111c6f88d728e7fb886cd3c8e9f8979d01e8"}
Jul 15 11:37:13.351838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.351572    2565 scope.go:117] "RemoveContainer" containerID="9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"
Jul 15 11:37:13.361972 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.361943    2565 scope.go:117] "RemoveContainer" containerID="6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"
Jul 15 11:37:13.370878 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.370859    2565 scope.go:117] "RemoveContainer" containerID="9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"
Jul 15 11:37:13.371405 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:37:13.371282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c\": container with ID starting with 9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c not found: ID does not exist" containerID="9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"
Jul 15 11:37:13.371405 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.371328    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c"} err="failed to get container status \"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c\": rpc error: code = NotFound desc = could not find container \"9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c\": container with ID starting with 9e8680616194fdf3846a78ff7ae0d8fc70a84f058a85927c8dabde9c07a47a8c not found: ID does not exist"
Jul 15 11:37:13.371405 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.371355    2565 scope.go:117] "RemoveContainer" containerID="6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"
Jul 15 11:37:13.371874 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:37:13.371724    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac\": container with ID starting with 6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac not found: ID does not exist" containerID="6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"
Jul 15 11:37:13.371874 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.371769    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac"} err="failed to get container status \"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac\": rpc error: code = NotFound desc = could not find container \"6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac\": container with ID starting with 6949221a4a80818372324c7cf56d4f9829f892092cccd1605d767eaec5e041ac not found: ID does not exist"
Jul 15 11:37:13.372486 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.372466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:37:13.375855 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.375833    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2"]
Jul 15 11:37:13.421640 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:13.421600    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:37:14.052989 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:14.052952    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:37:14.675431 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:14.675401    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" path="/var/lib/kubelet/pods/c55e24b0-7b89-402b-a500-127cfdb798b5/volumes"
Jul 15 11:37:24.053859 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:24.053811    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:37:28.446444 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:28.446371    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="main" containerID="cri-o://e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048" gracePeriod=45
Jul 15 11:37:29.207451 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.207425    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:37:29.318930 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.318836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.318930 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.318897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319209 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.318977    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319209 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319209 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmx6z\" (UniqueName: \"kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319209 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319209 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319142    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm\") pod \"3e720bd5-2127-421f-8af4-8bc510d75aa1\" (UID: \"3e720bd5-2127-421f-8af4-8bc510d75aa1\") "
Jul 15 11:37:29.319462 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:29.319528 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.319473    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.320156 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.320127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home" (OuterVolumeSpecName: "home") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:29.321689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.321658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm" (OuterVolumeSpecName: "dshm") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:29.321689 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.321668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:37:29.321835 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.321773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z" (OuterVolumeSpecName: "kube-api-access-vmx6z") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "kube-api-access-vmx6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:37:29.330914 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.330886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:29.391031 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.390974    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e720bd5-2127-421f-8af4-8bc510d75aa1" (UID: "3e720bd5-2127-421f-8af4-8bc510d75aa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:29.420178 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420143    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.420178 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420174    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e720bd5-2127-421f-8af4-8bc510d75aa1-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.420178 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420185    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmx6z\" (UniqueName: \"kubernetes.io/projected/3e720bd5-2127-421f-8af4-8bc510d75aa1-kube-api-access-vmx6z\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.420405 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420193    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.420405 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420205    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.420405 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.420218    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e720bd5-2127-421f-8af4-8bc510d75aa1-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:37:29.460511 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.460477    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerID="e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048" exitCode=0
Jul 15 11:37:29.460979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.460552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerDied","Data":"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"}
Jul 15 11:37:29.460979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.460597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3e720bd5-2127-421f-8af4-8bc510d75aa1","Type":"ContainerDied","Data":"f41e9e0c34382efa81dcacf11c14ef75acf7152f2c116448e10e9a633191a4ca"}
Jul 15 11:37:29.460979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.460604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 15 11:37:29.460979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.460620    2565 scope.go:117] "RemoveContainer" containerID="e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"
Jul 15 11:37:29.471081 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.471060    2565 scope.go:117] "RemoveContainer" containerID="06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"
Jul 15 11:37:29.479801 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.479782    2565 scope.go:117] "RemoveContainer" containerID="e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"
Jul 15 11:37:29.480120 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:37:29.480076    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048\": container with ID starting with e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048 not found: ID does not exist" containerID="e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"
Jul 15 11:37:29.480120 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.480101    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048"} err="failed to get container status \"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048\": rpc error: code = NotFound desc = could not find container \"e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048\": container with ID starting with e9a6506de1de0ee87e093af207e03d0f0def177a52af190061387d876ec4e048 not found: ID does not exist"
Jul 15 11:37:29.480120 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.480116    2565 scope.go:117] "RemoveContainer" containerID="06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"
Jul 15 11:37:29.480336 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:37:29.480317    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc\": container with ID starting with 06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc not found: ID does not exist" containerID="06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"
Jul 15 11:37:29.480376 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.480341    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc"} err="failed to get container status \"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc\": rpc error: code = NotFound desc = could not find container \"06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc\": container with ID starting with 06547c16b352594e103bbfe59cbe57240542ca781ae5844de8885af6c39a58bc not found: ID does not exist"
Jul 15 11:37:29.484716 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.484683    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:37:29.488230 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:29.488204    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 15 11:37:30.674246 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:30.674211    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" path="/var/lib/kubelet/pods/3e720bd5-2127-421f-8af4-8bc510d75aa1/volumes"
Jul 15 11:37:34.053193 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:34.053150    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:37:36.589792 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:36.589756    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:37:36.956950 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:36.956848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:37:44.053488 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:44.053442    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:37:44.675942 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:44.675899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:37:54.053877 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:37:54.053828    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:04.053109 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:04.053032    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:10.146728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:10.146695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:11.619276 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:11.619233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:11.692957 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:11.692913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:14.053725 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:14.053684    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:24.054011 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:24.053967    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:34.053949 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:34.053903    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:38.211994 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:38.211960    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:38.587441 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:38.587400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:44.053063 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:44.053005    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:38:49.152140 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:49.152099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:54.063531 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:54.063493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:38:54.071480 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:54.071453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:38:55.161278 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:55.161245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:38:55.169807 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:55.169773    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:38:55.296253 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:38:55.296224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:39:11.006923 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.006858    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" containerID="cri-o://49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d" gracePeriod=45
Jul 15 11:39:11.064568 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.064524    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 15 11:39:11.764727 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.764691    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:39:11.865856 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865924    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mslxs\" (UniqueName: \"kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.865990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866087 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.866064    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm\") pod \"a627cd05-1287-4b57-94c6-205f35050486\" (UID: \"a627cd05-1287-4b57-94c6-205f35050486\") "
Jul 15 11:39:11.866416 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.866384    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache" (OuterVolumeSpecName: "model-cache") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:11.866512 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.866485    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home" (OuterVolumeSpecName: "home") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:11.868203 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.868179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:39:11.868460 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.868436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs" (OuterVolumeSpecName: "kube-api-access-mslxs") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "kube-api-access-mslxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:39:11.868804 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.868783    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm" (OuterVolumeSpecName: "dshm") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:11.880245 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.880218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:11.927618 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.927542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a627cd05-1287-4b57-94c6-205f35050486" (UID: "a627cd05-1287-4b57-94c6-205f35050486"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:11.967058 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967013    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967068    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627cd05-1287-4b57-94c6-205f35050486-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967081    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967090    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967099    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslxs\" (UniqueName: \"kubernetes.io/projected/a627cd05-1287-4b57-94c6-205f35050486-kube-api-access-mslxs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967107    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.967221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:11.967115    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627cd05-1287-4b57-94c6-205f35050486-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 11:39:12.069298 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.069265    2565 generic.go:356] "Generic (PLEG): container finished" podID="a627cd05-1287-4b57-94c6-205f35050486" containerID="49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d" exitCode=0
Jul 15 11:39:12.069692 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.069320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"
Jul 15 11:39:12.069692 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.069360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerDied","Data":"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"}
Jul 15 11:39:12.069692 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.069419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn" event={"ID":"a627cd05-1287-4b57-94c6-205f35050486","Type":"ContainerDied","Data":"a1f9650fcd6a7abe7179c090f8a3031d17c1af4ebbd837b99ea7c647f515ab7b"}
Jul 15 11:39:12.069692 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.069443    2565 scope.go:117] "RemoveContainer" containerID="49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"
Jul 15 11:39:12.079086 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.079068    2565 scope.go:117] "RemoveContainer" containerID="6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"
Jul 15 11:39:12.087368 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.087348    2565 scope.go:117] "RemoveContainer" containerID="49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"
Jul 15 11:39:12.087604 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:39:12.087577    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d\": container with ID starting with 49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d not found: ID does not exist" containerID="49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"
Jul 15 11:39:12.087660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.087606    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d"} err="failed to get container status \"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d\": rpc error: code = NotFound desc = could not find container \"49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d\": container with ID starting with 49f5ddf8e80f3173db4e2e5bc7920ce58364ad7eb18f9ef765f203972fd0396d not found: ID does not exist"
Jul 15 11:39:12.087660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.087626    2565 scope.go:117] "RemoveContainer" containerID="6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"
Jul 15 11:39:12.087856 ip-10-0-137-92 kubenswrapper[2565]: E0715 11:39:12.087837    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612\": container with ID starting with 6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612 not found: ID does not exist" containerID="6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"
Jul 15 11:39:12.087911 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.087865    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612"} err="failed to get container status \"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612\": rpc error: code = NotFound desc = could not find container \"6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612\": container with ID starting with 6ba7b54561fae23029243ec44efd9a60357b18653fc8b7b236bc6350b1b5a612 not found: ID does not exist"
Jul 15 11:39:12.092208 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.092185    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:39:12.099173 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.099152    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn"]
Jul 15 11:39:12.674613 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:12.674579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a627cd05-1287-4b57-94c6-205f35050486" path="/var/lib/kubelet/pods/a627cd05-1287-4b57-94c6-205f35050486/volumes"
Jul 15 11:39:29.360232 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:39:29.360149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:40:07.866271 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:07.866238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:40:34.911728 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:34.911697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:40:34.915874 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:34.915853    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:40:34.924776 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:34.924753    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:40:34.930203 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:34.930184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:40:43.197254 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:43.197217    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:40:43.255745 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:43.255706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:40:52.544951 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:52.544914    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:40:52.645177 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:40:52.645136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:41:24.345373 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:41:24.345328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:41:24.685531 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:41:24.685271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:41:33.077584 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:41:33.077554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:41:59.114879 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:41:59.114841    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:41:59.186549 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:41:59.186517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:42:31.145472 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:42:31.145390    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:42:31.553330 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:42:31.553248    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:42:35.841809 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:42:35.841777    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:43:04.889797 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:43:04.889755    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:43:42.959640 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:43:42.959597    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:43:49.620678 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:43:49.620636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:44:23.945026 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.944994    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945385    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945397    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945405    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945411    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945419    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="storage-initializer"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="storage-initializer"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945434    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="storage-initializer"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="storage-initializer"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945454    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945469    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="storage-initializer"
Jul 15 11:44:23.945500 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945474    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="storage-initializer"
Jul 15 11:44:23.945918 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945531    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd05-1287-4b57-94c6-205f35050486" containerName="main"
Jul 15 11:44:23.945918 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945539    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55e24b0-7b89-402b-a500-127cfdb798b5" containerName="main"
Jul 15 11:44:23.945918 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.945545    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e720bd5-2127-421f-8af4-8bc510d75aa1" containerName="main"
Jul 15 11:44:23.948733 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.948715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:23.952977 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.952957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:44:23.953321 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.953306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:44:23.953883 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.953869    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 15 11:44:23.953979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.953906    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 15 11:44:23.994838 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:23.994801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 11:44:24.085215 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.085173    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:44:24.094142 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094304 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094304 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094421 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094421 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094421 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwxz\" (UniqueName: \"kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.094571 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.094488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195658 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195658 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195658 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195927 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwxz\" (UniqueName: \"kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195927 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195927 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195927 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.195927 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.195889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.196221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.196070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.196221 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.196107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.196339 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.196283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.196339 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.196315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.196580 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.196562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.198225 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.198209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.198510 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.198492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.221983 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.221962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwxz\" (UniqueName: \"kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz\") pod \"lora-multiple-adapters-test-kserve-58f99768fc-94c45\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.258979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.258955    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:24.405901 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.405877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 11:44:24.406991 ip-10-0-137-92 kubenswrapper[2565]: W0715 11:44:24.406967    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe1e13_a5cf_47fc_8af7_a119ea343ab8.slice/crio-a9eff61796a3fab76e0e155d1ce5badeb8e834415963e78fdc4309a328f3d311 WatchSource:0}: Error finding container a9eff61796a3fab76e0e155d1ce5badeb8e834415963e78fdc4309a328f3d311: Status 404 returned error can't find the container with id a9eff61796a3fab76e0e155d1ce5badeb8e834415963e78fdc4309a328f3d311
Jul 15 11:44:24.763378 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.763287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerStarted","Data":"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"}
Jul 15 11:44:24.763378 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:24.763358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerStarted","Data":"a9eff61796a3fab76e0e155d1ce5badeb8e834415963e78fdc4309a328f3d311"}
Jul 15 11:44:25.726275 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:25.726240    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:44:25.856022 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:25.855974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:44:25.973742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:25.973709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:44:28.792162 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:28.792034    2565 generic.go:356] "Generic (PLEG): container finished" podID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerID="21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35" exitCode=0
Jul 15 11:44:28.792162 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:28.792111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerDied","Data":"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"}
Jul 15 11:44:29.802310 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:29.802282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerStarted","Data":"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"}
Jul 15 11:44:29.827429 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:29.827371    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podStartSLOduration=2.976177843 podStartE2EDuration="6.827351538s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.851173695s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:44:29.8259882 +0000 UTC m=+3535.660732651" watchObservedRunningTime="2026-07-15 11:44:29.827351538 +0000 UTC m=+3535.662095988"
Jul 15 11:44:30.259858 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:30.259817    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:34.259978 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:34.259939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:44:34.261651 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:34.261625    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:44:44.260315 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:44.260273    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:44:54.259848 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:44:54.259806    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:45:00.030830 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:00.030788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:00.317940 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:00.317849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:04.260255 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:04.260209    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:45:12.615416 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:12.615380    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:14.260219 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:14.260178    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:45:24.260217 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:24.260169    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:45:33.799116 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:33.799076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:34.259991 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:34.259943    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 15 11:45:34.951901 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:34.951874    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:45:34.956021 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:34.956002    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:45:34.965796 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:34.965774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:45:34.969793 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:34.969776    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:45:36.361351 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:36.361316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:44.269506 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:44.269473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:45:44.277284 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:44.277253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 11:45:59.120316 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:59.120272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:45:59.182204 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:45:59.182163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:46:06.131183 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:46:06.131145    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:46:41.697947 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:46:41.697904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:47:38.086188 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:47:38.086145    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:48:14.382182 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:48:14.382143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:50:34.990742 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:50:34.990697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:50:34.994898 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:50:34.994877    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:50:35.005233 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:50:35.005216    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:50:35.008979 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:50:35.008958    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:53:41.566343 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:53:41.566306    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:54:22.366760 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:54:22.366727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:54:22.467660 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:54:22.467623    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:54:34.373428 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:54:34.373356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:54:34.548697 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:54:34.548634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:55:04.025624 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:04.025587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:55:04.312324 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:04.312234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:55:11.204654 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:11.204608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:55:35.026577 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:35.026546    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:55:35.030734 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:35.030710    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:55:35.045807 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:35.045783    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 11:55:35.049663 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:55:35.049645    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 11:56:38.699864 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:56:38.699816    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:56:38.788520 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:56:38.788489    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:56:51.495069 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:56:51.495009    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:57:16.937161 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:57:16.937127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:57:17.512098 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:57:17.512061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 11:57:27.261693 ip-10-0-137-92 kubenswrapper[2565]: I0715 11:57:27.261611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:00:06.064484 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:00:06.064448    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:00:35.063464 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:00:35.063381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 12:00:35.067623 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:00:35.067597    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 12:00:35.090576 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:00:35.090547    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 12:00:35.094340 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:00:35.094320    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 12:01:06.152674 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:06.152637    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:01:06.226779 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:06.226742    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:01:32.724949 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:32.724913    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 12:01:32.756017 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:32.755984    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:01:47.750529 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:47.750470    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" containerID="cri-o://dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3" gracePeriod=45
Jul 15 12:01:48.336761 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.336736    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 12:01:48.400910 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.400910 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.400910 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtwxz\" (UniqueName: \"kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.400910 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400901    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.401307 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.401307 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.400954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.401307 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.401307 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401084    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm\") pod \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\" (UID: \"4efe1e13-a5cf-47fc-8af7-a119ea343ab8\") "
Jul 15 12:01:48.401307 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:01:48.401565 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401330    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.401565 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home" (OuterVolumeSpecName: "home") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:01:48.401565 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.401432    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 12:01:48.404062 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.404011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:01:48.404184 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.404061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz" (OuterVolumeSpecName: "kube-api-access-gtwxz") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "kube-api-access-gtwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:01:48.404184 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.404095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm" (OuterVolumeSpecName: "dshm") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:01:48.418141 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.418103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:01:48.470986 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.470943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4efe1e13-a5cf-47fc-8af7-a119ea343ab8" (UID: "4efe1e13-a5cf-47fc-8af7-a119ea343ab8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:01:48.501988 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.501958    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.501988 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.501982    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.501988 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.501992    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.502246 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.502002    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwxz\" (UniqueName: \"kubernetes.io/projected/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-kube-api-access-gtwxz\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.502246 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.502010    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.502246 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.502018    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.502246 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.502025    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4efe1e13-a5cf-47fc-8af7-a119ea343ab8-cabundle-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:01:48.631782 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.631747    2565 generic.go:356] "Generic (PLEG): container finished" podID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerID="dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3" exitCode=0
Jul 15 12:01:48.631980 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.631807    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"
Jul 15 12:01:48.631980 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.631811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerDied","Data":"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"}
Jul 15 12:01:48.631980 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.631864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" event={"ID":"4efe1e13-a5cf-47fc-8af7-a119ea343ab8","Type":"ContainerDied","Data":"a9eff61796a3fab76e0e155d1ce5badeb8e834415963e78fdc4309a328f3d311"}
Jul 15 12:01:48.631980 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.631885    2565 scope.go:117] "RemoveContainer" containerID="dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"
Jul 15 12:01:48.641879 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.641862    2565 scope.go:117] "RemoveContainer" containerID="21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"
Jul 15 12:01:48.650737 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.650718    2565 scope.go:117] "RemoveContainer" containerID="dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"
Jul 15 12:01:48.651010 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:01:48.650969    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3\": container with ID starting with dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3 not found: ID does not exist" containerID="dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"
Jul 15 12:01:48.651132 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.651005    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3"} err="failed to get container status \"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3\": rpc error: code = NotFound desc = could not find container \"dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3\": container with ID starting with dbfc7c48684fb95e7567637ece0ae3eaf1628b27b4aa339c46523ebc235d2cd3 not found: ID does not exist"
Jul 15 12:01:48.651132 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.651024    2565 scope.go:117] "RemoveContainer" containerID="21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"
Jul 15 12:01:48.651283 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:01:48.651265    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35\": container with ID starting with 21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35 not found: ID does not exist" containerID="21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"
Jul 15 12:01:48.651324 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.651291    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35"} err="failed to get container status \"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35\": rpc error: code = NotFound desc = could not find container \"21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35\": container with ID starting with 21c02d3768935f9b649764ee30f0e2c42dac309de89d866c7eaaba28c924bf35 not found: ID does not exist"
Jul 15 12:01:48.656745 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.656721    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 12:01:48.662673 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.662650    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45"]
Jul 15 12:01:48.674337 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:48.674302    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" path="/var/lib/kubelet/pods/4efe1e13-a5cf-47fc-8af7-a119ea343ab8/volumes"
Jul 15 12:01:49.270507 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:49.270456    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 15 12:01:59.833958 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.833903    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:01:59.834539 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.834515    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="storage-initializer"
Jul 15 12:01:59.834608 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.834546    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="storage-initializer"
Jul 15 12:01:59.834608 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.834558    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main"
Jul 15 12:01:59.834608 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.834567    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main"
Jul 15 12:01:59.834741 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.834668    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efe1e13-a5cf-47fc-8af7-a119ea343ab8" containerName="main"
Jul 15 12:01:59.837004 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.836985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.871804 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.871772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 12:01:59.872582 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.872558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 12:01:59.872673 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.872559    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 15 12:01:59.881665 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.881638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:01:59.902203 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902385 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902385 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvbp\" (UniqueName: \"kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902385 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902563 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902563 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.902563 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.902508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:01:59.911288 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:01:59.911254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:00.003031 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvbp\" (UniqueName: \"kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003430 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003430 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003430 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003590 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.003805 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.004223 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.004223 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.003971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.005890 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.005869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.006004 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.005914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.013857 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.013831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvbp\" (UniqueName: \"kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp\") pod \"tls-verification-test-kserve-df55b4f65-htl9f\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.146974 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.146888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:00.301017 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.300985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:02:00.304068 ip-10-0-137-92 kubenswrapper[2565]: W0715 12:02:00.304025    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aea78a_1030_4a6a_9d21_23a90da34f94.slice/crio-a07d65bbb960bbd44a0c7c86b2f3687c6b85244013106360d7f2185c63cf18bb WatchSource:0}: Error finding container a07d65bbb960bbd44a0c7c86b2f3687c6b85244013106360d7f2185c63cf18bb: Status 404 returned error can't find the container with id a07d65bbb960bbd44a0c7c86b2f3687c6b85244013106360d7f2185c63cf18bb
Jul 15 12:02:00.432475 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.432392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:00.701066 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.700932    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerStarted","Data":"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"}
Jul 15 12:02:00.701066 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:00.700992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerStarted","Data":"a07d65bbb960bbd44a0c7c86b2f3687c6b85244013106360d7f2185c63cf18bb"}
Jul 15 12:02:03.721762 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:03.721657    2565 generic.go:356] "Generic (PLEG): container finished" podID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerID="53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8" exitCode=0
Jul 15 12:02:03.721762 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:03.721737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerDied","Data":"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"}
Jul 15 12:02:04.730274 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:04.730233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerStarted","Data":"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"}
Jul 15 12:02:04.761488 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:04.761435    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podStartSLOduration=2.675797915 podStartE2EDuration="5.761418577s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.085620662s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:02:04.756381202 +0000 UTC m=+4590.591125662" watchObservedRunningTime="2026-07-15 12:02:04.761418577 +0000 UTC m=+4590.596163026"
Jul 15 12:02:05.147194 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:05.147159    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:05.339400 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:05.339365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:05.417287 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:05.417204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:06.027442 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:06.027400    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 12:02:06.027937 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:06.027599    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" podUID="751a12f4-9aaf-44f8-9c81-b3c223db455e" containerName="manager" containerID="cri-o://a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d" gracePeriod=30
Jul 15 12:02:07.391921 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.391888    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 12:02:07.475686 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.475588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfvgj\" (UniqueName: \"kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj\") pod \"751a12f4-9aaf-44f8-9c81-b3c223db455e\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") "
Jul 15 12:02:07.475864 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.475735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert\") pod \"751a12f4-9aaf-44f8-9c81-b3c223db455e\" (UID: \"751a12f4-9aaf-44f8-9c81-b3c223db455e\") "
Jul 15 12:02:07.478024 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.477990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj" (OuterVolumeSpecName: "kube-api-access-wfvgj") pod "751a12f4-9aaf-44f8-9c81-b3c223db455e" (UID: "751a12f4-9aaf-44f8-9c81-b3c223db455e"). InnerVolumeSpecName "kube-api-access-wfvgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:02:07.478199 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.478083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert" (OuterVolumeSpecName: "cert") pod "751a12f4-9aaf-44f8-9c81-b3c223db455e" (UID: "751a12f4-9aaf-44f8-9c81-b3c223db455e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:02:07.576677 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.576640    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751a12f4-9aaf-44f8-9c81-b3c223db455e-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:02:07.576677 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.576674    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvgj\" (UniqueName: \"kubernetes.io/projected/751a12f4-9aaf-44f8-9c81-b3c223db455e-kube-api-access-wfvgj\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:02:07.751924 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.751836    2565 generic.go:356] "Generic (PLEG): container finished" podID="751a12f4-9aaf-44f8-9c81-b3c223db455e" containerID="a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d" exitCode=0
Jul 15 12:02:07.751924 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.751903    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7"
Jul 15 12:02:07.752135 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.751929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" event={"ID":"751a12f4-9aaf-44f8-9c81-b3c223db455e","Type":"ContainerDied","Data":"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"}
Jul 15 12:02:07.752135 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.751991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dd9fc96f-r22s7" event={"ID":"751a12f4-9aaf-44f8-9c81-b3c223db455e","Type":"ContainerDied","Data":"44b6c292c19a27e43e0a0b777edeafff7fda5e1beac8456f2e1c71d179f8e376"}
Jul 15 12:02:07.752135 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.752017    2565 scope.go:117] "RemoveContainer" containerID="a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"
Jul 15 12:02:07.763154 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.763135    2565 scope.go:117] "RemoveContainer" containerID="a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"
Jul 15 12:02:07.763489 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:02:07.763469    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d\": container with ID starting with a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d not found: ID does not exist" containerID="a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"
Jul 15 12:02:07.763554 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.763501    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d"} err="failed to get container status \"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d\": rpc error: code = NotFound desc = could not find container \"a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d\": container with ID starting with a4ecc697944e1efd9ef4d44bf44b0cf705985e76a0ab9a0b3bdc78650ba63f8d not found: ID does not exist"
Jul 15 12:02:07.778856 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.778827    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 12:02:07.783948 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:07.783923    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-dd9fc96f-r22s7"]
Jul 15 12:02:08.674728 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:08.674691    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751a12f4-9aaf-44f8-9c81-b3c223db455e" path="/var/lib/kubelet/pods/751a12f4-9aaf-44f8-9c81-b3c223db455e/volumes"
Jul 15 12:02:10.147778 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:10.147739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:02:10.149570 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:10.149538    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:02:20.147382 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:20.147335    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:02:29.564001 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:29.563967    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:30.147629 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:30.147587    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:02:40.148270 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:40.148224    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:02:43.774677 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:43.774641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:44.139552 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:44.139513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:02:50.148324 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:50.148279    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:02:56.101591 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:02:56.101541    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:03:00.147860 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:00.147816    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:03:10.147971 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:10.147923    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:03:20.148192 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:20.148148    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:03:30.147977 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:30.147885    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:03:40.148325 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:40.148275    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:03:50.148255 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:03:50.148213    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 15 12:04:00.158031 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:00.158001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:04:00.165950 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:00.165923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:04:05.075839 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:05.075794    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:04:05.140983 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:05.140950    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:04:05.192355 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:05.192322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:04:20.165801 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:20.165757    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main" containerID="cri-o://828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38" gracePeriod=45
Jul 15 12:04:20.927525 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:20.927497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:04:21.048975 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.048887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.048975 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.048917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.048975 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.048951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.049317 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.049317 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.049317 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tvbp\" (UniqueName: \"kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.049317 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location\") pod \"90aea78a-1030-4a6a-9d21-23a90da34f94\" (UID: \"90aea78a-1030-4a6a-9d21-23a90da34f94\") "
Jul 15 12:04:21.049527 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:21.049807 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.049782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home" (OuterVolumeSpecName: "home") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:21.051547 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.051506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp" (OuterVolumeSpecName: "kube-api-access-2tvbp") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "kube-api-access-2tvbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:04:21.051694 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.051669    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm" (OuterVolumeSpecName: "dshm") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:21.051796 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.051779    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:04:21.062149 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.062126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:21.118555 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.118498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90aea78a-1030-4a6a-9d21-23a90da34f94" (UID: "90aea78a-1030-4a6a-9d21-23a90da34f94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:21.150027 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.149980    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aea78a-1030-4a6a-9d21-23a90da34f94-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150027 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150024    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150211 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150034    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvbp\" (UniqueName: \"kubernetes.io/projected/90aea78a-1030-4a6a-9d21-23a90da34f94-kube-api-access-2tvbp\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150211 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150066    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150211 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150076    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150211 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150084    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.150211 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.150091    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90aea78a-1030-4a6a-9d21-23a90da34f94-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:04:21.529634 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.529600    2565 generic.go:356] "Generic (PLEG): container finished" podID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerID="828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38" exitCode=0
Jul 15 12:04:21.530146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.529663    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"
Jul 15 12:04:21.530146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.529678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerDied","Data":"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"}
Jul 15 12:04:21.530146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.529739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f" event={"ID":"90aea78a-1030-4a6a-9d21-23a90da34f94","Type":"ContainerDied","Data":"a07d65bbb960bbd44a0c7c86b2f3687c6b85244013106360d7f2185c63cf18bb"}
Jul 15 12:04:21.530146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.529764    2565 scope.go:117] "RemoveContainer" containerID="828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"
Jul 15 12:04:21.542496 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.542477    2565 scope.go:117] "RemoveContainer" containerID="53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"
Jul 15 12:04:21.550772 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.550739    2565 scope.go:117] "RemoveContainer" containerID="828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"
Jul 15 12:04:21.551130 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:04:21.551104    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38\": container with ID starting with 828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38 not found: ID does not exist" containerID="828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"
Jul 15 12:04:21.551239 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.551143    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38"} err="failed to get container status \"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38\": rpc error: code = NotFound desc = could not find container \"828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38\": container with ID starting with 828f1c25311af6999325d4450d6546f8c3e2197de8643ce4751dacfb0393bb38 not found: ID does not exist"
Jul 15 12:04:21.551239 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.551166    2565 scope.go:117] "RemoveContainer" containerID="53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"
Jul 15 12:04:21.551517 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:04:21.551490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8\": container with ID starting with 53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8 not found: ID does not exist" containerID="53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"
Jul 15 12:04:21.551605 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.551533    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8"} err="failed to get container status \"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8\": rpc error: code = NotFound desc = could not find container \"53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8\": container with ID starting with 53d83152b0471589dc22d4b68d412e643ed2aad857f878c56191c4fc94072bd8 not found: ID does not exist"
Jul 15 12:04:21.554116 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.554093    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:04:21.557579 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:21.557560    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f"]
Jul 15 12:04:22.674142 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:22.674111    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" path="/var/lib/kubelet/pods/90aea78a-1030-4a6a-9d21-23a90da34f94/volumes"
Jul 15 12:04:47.223077 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223020    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223422    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751a12f4-9aaf-44f8-9c81-b3c223db455e" containerName="manager"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223464    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="751a12f4-9aaf-44f8-9c81-b3c223db455e" containerName="manager"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223473    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="storage-initializer"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223479    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="storage-initializer"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223531    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90aea78a-1030-4a6a-9d21-23a90da34f94" containerName="main"
Jul 15 12:04:47.223568 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.223539    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="751a12f4-9aaf-44f8-9c81-b3c223db455e" containerName="manager"
Jul 15 12:04:47.227903 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.227886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.231198 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.231174    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 12:04:47.231753 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.231722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 12:04:47.231867 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.231823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 15 12:04:47.241373 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241495 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241495 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241597 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241597 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwdz\" (UniqueName: \"kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241716 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.241716 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.241661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.244889 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.244865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:04:47.342837 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.342995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343070 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.343068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwdz\" (UniqueName: \"kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343386 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.343343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343513 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.343490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343571 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.343532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.343571 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.343542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.345451 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.345429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.345694 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.345674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.352374 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.352349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwdz\" (UniqueName: \"kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz\") pod \"prestop-hook-test-kserve-85ccc87949-mv9l4\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.538524 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.538433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:47.686456 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:47.686427    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:04:47.687496 ip-10-0-137-92 kubenswrapper[2565]: W0715 12:04:47.687470    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bb7254_c346_4f40_89f9_3170eefd64ab.slice/crio-dc3f7a840498cb1c1cbb0b29fbefd0b8453659da9464e220df0a844b78939e8f WatchSource:0}: Error finding container dc3f7a840498cb1c1cbb0b29fbefd0b8453659da9464e220df0a844b78939e8f: Status 404 returned error can't find the container with id dc3f7a840498cb1c1cbb0b29fbefd0b8453659da9464e220df0a844b78939e8f
Jul 15 12:04:48.677567 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:48.677526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerStarted","Data":"7bad0d136834d211a5b5cebdde6898b7864693a0f1fb3d7fd1d292328984c300"}
Jul 15 12:04:48.678071 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:48.677594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerStarted","Data":"dc3f7a840498cb1c1cbb0b29fbefd0b8453659da9464e220df0a844b78939e8f"}
Jul 15 12:04:51.699817 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:51.699775    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerID="7bad0d136834d211a5b5cebdde6898b7864693a0f1fb3d7fd1d292328984c300" exitCode=0
Jul 15 12:04:51.700259 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:51.699853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerDied","Data":"7bad0d136834d211a5b5cebdde6898b7864693a0f1fb3d7fd1d292328984c300"}
Jul 15 12:04:52.706595 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:52.706560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerStarted","Data":"e0f35b351b68815fec0d6451a53e28b67be29226094f5da6e6968dc060a04e73"}
Jul 15 12:04:52.731184 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:52.731128    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" podStartSLOduration=2.676237114 podStartE2EDuration="5.731110505s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.054873391s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:04:52.72803552 +0000 UTC m=+4758.562779989" watchObservedRunningTime="2026-07-15 12:04:52.731110505 +0000 UTC m=+4758.565854955"
Jul 15 12:04:53.538786 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:53.538747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:57.539465 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:57.539370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:04:57.541245 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:04:57.541218    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 15 12:05:03.519688 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:03.519652    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:05:03.529707 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:03.529668    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:05:03.598198 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:03.598159    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:05:18.547901 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:18.547833    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="main" containerID="cri-o://e0f35b351b68815fec0d6451a53e28b67be29226094f5da6e6968dc060a04e73" gracePeriod=45
Jul 15 12:05:35.099882 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:35.099852    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 12:05:35.104259 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:35.104237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 12:05:35.130889 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:35.130860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 12:05:35.135329 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:35.135303    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bg9qp_22f1ccaa-f805-44bd-a129-148ad7244245/ovn-acl-logging/0.log"
Jul 15 12:05:35.734391 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:35.734353    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:05:57.622978 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:05:57.622940    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:04.114910 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.114871    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-85ccc87949-mv9l4_a8bb7254-c346-4f40-89f9-3170eefd64ab/main/0.log"
Jul 15 12:06:04.115365 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.115261    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerID="e0f35b351b68815fec0d6451a53e28b67be29226094f5da6e6968dc060a04e73" exitCode=137
Jul 15 12:06:04.115365 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.115338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerDied","Data":"e0f35b351b68815fec0d6451a53e28b67be29226094f5da6e6968dc060a04e73"}
Jul 15 12:06:04.198865 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.198846    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-85ccc87949-mv9l4_a8bb7254-c346-4f40-89f9-3170eefd64ab/main/0.log"
Jul 15 12:06:04.200560 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.200543    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:06:04.270712 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270769    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvwdz\" (UniqueName: \"kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.270880 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.270851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache\") pod \"a8bb7254-c346-4f40-89f9-3170eefd64ab\" (UID: \"a8bb7254-c346-4f40-89f9-3170eefd64ab\") "
Jul 15 12:06:04.271218 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.271095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:06:04.271345 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.271318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home" (OuterVolumeSpecName: "home") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:06:04.271345 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.271325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:06:04.273197 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.273173    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:06:04.273631 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.273608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm" (OuterVolumeSpecName: "dshm") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:06:04.273701 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.273614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz" (OuterVolumeSpecName: "kube-api-access-fvwdz") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "kube-api-access-fvwdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:06:04.341599 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.341553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8bb7254-c346-4f40-89f9-3170eefd64ab" (UID: "a8bb7254-c346-4f40-89f9-3170eefd64ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:06:04.372119 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372093    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bb7254-c346-4f40-89f9-3170eefd64ab-tls-certs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372119 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372117    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-kserve-provision-location\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372310 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372127    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-home\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372310 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372136    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-dshm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372310 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372143    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwdz\" (UniqueName: \"kubernetes.io/projected/a8bb7254-c346-4f40-89f9-3170eefd64ab-kube-api-access-fvwdz\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372310 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372153    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-model-cache\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:04.372310 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:04.372166    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bb7254-c346-4f40-89f9-3170eefd64ab-tmp-dir\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:06:05.121716 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.121683    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-85ccc87949-mv9l4_a8bb7254-c346-4f40-89f9-3170eefd64ab/main/0.log"
Jul 15 12:06:05.122146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.122035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4" event={"ID":"a8bb7254-c346-4f40-89f9-3170eefd64ab","Type":"ContainerDied","Data":"dc3f7a840498cb1c1cbb0b29fbefd0b8453659da9464e220df0a844b78939e8f"}
Jul 15 12:06:05.122146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.122103    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"
Jul 15 12:06:05.122229 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.122110    2565 scope.go:117] "RemoveContainer" containerID="e0f35b351b68815fec0d6451a53e28b67be29226094f5da6e6968dc060a04e73"
Jul 15 12:06:05.131502 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.131485    2565 scope.go:117] "RemoveContainer" containerID="7bad0d136834d211a5b5cebdde6898b7864693a0f1fb3d7fd1d292328984c300"
Jul 15 12:06:05.143480 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.143459    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:06:05.146972 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:05.146946    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4"]
Jul 15 12:06:06.674812 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:06.674778    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" path="/var/lib/kubelet/pods/a8bb7254-c346-4f40-89f9-3170eefd64ab/volumes"
Jul 15 12:06:11.412779 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.412724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zx6sn/must-gather-xk7tb"]
Jul 15 12:06:11.413219 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.413188    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="main"
Jul 15 12:06:11.413219 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.413204    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="main"
Jul 15 12:06:11.413295 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.413229    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="storage-initializer"
Jul 15 12:06:11.413295 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.413236    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="storage-initializer"
Jul 15 12:06:11.413295 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.413289    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bb7254-c346-4f40-89f9-3170eefd64ab" containerName="main"
Jul 15 12:06:11.416490 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.416474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.420658 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.420636    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx6sn\"/\"kube-root-ca.crt\""
Jul 15 12:06:11.421186 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.421167    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx6sn\"/\"openshift-service-ca.crt\""
Jul 15 12:06:11.428643 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.428619    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx6sn/must-gather-xk7tb"]
Jul 15 12:06:11.436221 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.436192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w975x\" (UniqueName: \"kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.436352 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.436232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.441374 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.441304    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:11.537560 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.537521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.537733 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.537643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w975x\" (UniqueName: \"kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.537923 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.537903    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.541878 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.541853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:11.550208 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.550187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w975x\" (UniqueName: \"kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x\") pod \"must-gather-xk7tb\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") " pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.737732 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.737639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:06:11.893304 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.893266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx6sn/must-gather-xk7tb"]
Jul 15 12:06:11.896867 ip-10-0-137-92 kubenswrapper[2565]: W0715 12:06:11.896828    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4964f966_0dee_4929_9ccc_cc3361a9535e.slice/crio-7b6cd2d290d779d1461c98dff5d7a2b438ba41188ab693aa9186200833c82c24 WatchSource:0}: Error finding container 7b6cd2d290d779d1461c98dff5d7a2b438ba41188ab693aa9186200833c82c24: Status 404 returned error can't find the container with id 7b6cd2d290d779d1461c98dff5d7a2b438ba41188ab693aa9186200833c82c24
Jul 15 12:06:11.898523 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:11.898497    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 12:06:12.159367 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:12.159336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" event={"ID":"4964f966-0dee-4929-9ccc-cc3361a9535e","Type":"ContainerStarted","Data":"7b6cd2d290d779d1461c98dff5d7a2b438ba41188ab693aa9186200833c82c24"}
Jul 15 12:06:18.208506 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:18.208466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" event={"ID":"4964f966-0dee-4929-9ccc-cc3361a9535e","Type":"ContainerStarted","Data":"2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89"}
Jul 15 12:06:18.208888 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:18.208519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" event={"ID":"4964f966-0dee-4929-9ccc-cc3361a9535e","Type":"ContainerStarted","Data":"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"}
Jul 15 12:06:18.230542 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:18.230490    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" podStartSLOduration=1.287847495 podStartE2EDuration="7.230474145s" totalImagesPullingTime="5.94262665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:06:18.227627976 +0000 UTC m=+4844.062372425" watchObservedRunningTime="2026-07-15 12:06:18.230474145 +0000 UTC m=+4844.065218594"
Jul 15 12:06:27.472321 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:27.472246    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:27.541648 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:27.541615    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:27.541978 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:27.541962    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:28.984263 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:28.984220    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:29.020206 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:29.020174    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:29.020445 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:29.020427    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:30.474627 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:30.474600    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:30.507431 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:30.507399    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:30.507683 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:30.507670    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:31.895064 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:31.895003    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:31.928172 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:31.928146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:31.928476 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:31.928460    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:33.352452 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:33.352403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:33.388088 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:33.388061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:33.388363 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:33.388344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:34.811866 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:34.811836    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:34.847780 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:34.847747    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:34.848063 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:34.848021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:36.347193 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:36.347165    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:36.388447 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:36.388417    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:36.388856 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:36.388838    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:37.872584 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:37.872555    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:38.059248 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:38.059211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:38.059461 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:38.059445    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:39.486495 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:39.486469    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:39.529237 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:39.529211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:39.529531 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:39.529516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:40.946951 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:40.946898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:40.981534 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:40.981509    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:40.981786 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:40.981768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:42.354700 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:42.354666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:42.392179 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:42.392144    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:42.392525 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:42.392507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:43.785090 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:43.785053    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:43.817299 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:43.817261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:43.817667 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:43.817650    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:45.209146 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:45.209121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:45.245542 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:45.245515    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:45.245709 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:45.245702    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:46.642449 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:46.642422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:46.678478 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:46.678451    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:46.678844 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:46.678824    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:48.099238 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:48.099207    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:48.135677 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:48.135647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:48.135952 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:48.135933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:49.593204 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:49.593164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-qm8ll_a1313084-7836-49b9-a6e0-a7ccb413d04a/storage-initializer/0.log"
Jul 15 12:06:49.639551 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:49.639514    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/1.log"
Jul 15 12:06:49.639897 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:49.639879    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-65d6b86cbf-n2jmt_9aacc01e-5225-4093-8788-5cd9b555b523/istio-proxy/0.log"
Jul 15 12:06:52.018685 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:52.018654    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mzhsm_435a8d23-fa77-414b-be7f-0dea1442a91a/discovery/0.log"
Jul 15 12:06:52.084896 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:52.084867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4b849546-cmq8n_aea22a48-4abf-4d49-9058-99d9ad240187/router/0.log"
Jul 15 12:06:53.977755 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:53.977722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mzhsm_435a8d23-fa77-414b-be7f-0dea1442a91a/discovery/0.log"
Jul 15 12:06:54.068730 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:54.068695    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4b849546-cmq8n_aea22a48-4abf-4d49-9058-99d9ad240187/router/0.log"
Jul 15 12:06:55.051282 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:55.051247    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-45x42_2a1022a7-a38b-422a-a575-7d634cb767a8/manager/0.log"
Jul 15 12:06:55.065872 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:55.065844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-gc6m6_4d35bccc-aa92-4731-9f68-4ece273ee422/manager/0.log"
Jul 15 12:06:55.399462 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:55.399436    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-r4hbr_b7de55b9-da42-4bc9-be97-975ea925f74e/manager/0.log"
Jul 15 12:06:56.436577 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:56.436543    2565 generic.go:356] "Generic (PLEG): container finished" podID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerID="6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9" exitCode=0
Jul 15 12:06:56.437007 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:56.436617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" event={"ID":"4964f966-0dee-4929-9ccc-cc3361a9535e","Type":"ContainerDied","Data":"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"}
Jul 15 12:06:56.437007 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:56.436954    2565 scope.go:117] "RemoveContainer" containerID="6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"
Jul 15 12:06:57.220311 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:57.220272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zx6sn_must-gather-xk7tb_4964f966-0dee-4929-9ccc-cc3361a9535e/gather/0.log"
Jul 15 12:06:57.968051 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:57.968006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:58.169054 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:58.169019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:58.315985 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:58.315900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:06:59.346210 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:06:59.346179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:07:01.135685 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:01.135653    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lctcq_24527bdc-899a-498d-ae7c-8c09a08e921d/global-pull-secret-syncer/0.log"
Jul 15 12:07:01.181745 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:01.181717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log"
Jul 15 12:07:01.400613 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:01.400536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-92.ec2.internal_7f3e59195d56b4d838109b2802c151a5/haproxy/0.log"
Jul 15 12:07:02.962325 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:02.962290    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zx6sn/must-gather-xk7tb"]
Jul 15 12:07:02.962786 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:02.962483    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zx6sn/must-gather-xk7tb" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="copy" containerID="cri-o://2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89" gracePeriod=2
Jul 15 12:07:02.970059 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:02.970018    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zx6sn/must-gather-xk7tb"]
Jul 15 12:07:03.207927 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.207898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zx6sn_must-gather-xk7tb_4964f966-0dee-4929-9ccc-cc3361a9535e/copy/0.log"
Jul 15 12:07:03.209457 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.209439    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:03.211840 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:03.211810    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xk7tb\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zx6sn\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:03.311504 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.311408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w975x\" (UniqueName: \"kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x\") pod \"4964f966-0dee-4929-9ccc-cc3361a9535e\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") "
Jul 15 12:07:03.311504 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.311460    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output\") pod \"4964f966-0dee-4929-9ccc-cc3361a9535e\" (UID: \"4964f966-0dee-4929-9ccc-cc3361a9535e\") "
Jul 15 12:07:03.313813 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.313782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x" (OuterVolumeSpecName: "kube-api-access-w975x") pod "4964f966-0dee-4929-9ccc-cc3361a9535e" (UID: "4964f966-0dee-4929-9ccc-cc3361a9535e"). InnerVolumeSpecName "kube-api-access-w975x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:07:03.322455 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.322427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4964f966-0dee-4929-9ccc-cc3361a9535e" (UID: "4964f966-0dee-4929-9ccc-cc3361a9535e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:07:03.412502 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.412453    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w975x\" (UniqueName: \"kubernetes.io/projected/4964f966-0dee-4929-9ccc-cc3361a9535e-kube-api-access-w975x\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:07:03.412502 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.412496    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4964f966-0dee-4929-9ccc-cc3361a9535e-must-gather-output\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 15 12:07:03.475368 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.475338    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zx6sn_must-gather-xk7tb_4964f966-0dee-4929-9ccc-cc3361a9535e/copy/0.log"
Jul 15 12:07:03.475681 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.475652    2565 generic.go:356] "Generic (PLEG): container finished" podID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerID="2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89" exitCode=143
Jul 15 12:07:03.475779 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.475694    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:03.475779 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.475747    2565 scope.go:117] "RemoveContainer" containerID="2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89"
Jul 15 12:07:03.477925 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:03.477896    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xk7tb\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zx6sn\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:03.484926 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.484910    2565 scope.go:117] "RemoveContainer" containerID="6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"
Jul 15 12:07:03.485512 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:03.485491    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xk7tb\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zx6sn\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:03.498390 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.498372    2565 scope.go:117] "RemoveContainer" containerID="2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89"
Jul 15 12:07:03.498690 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:03.498667    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89\": container with ID starting with 2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89 not found: ID does not exist" containerID="2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89"
Jul 15 12:07:03.498796 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.498695    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89"} err="failed to get container status \"2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89\": rpc error: code = NotFound desc = could not find container \"2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89\": container with ID starting with 2315f56d7864cf0852d06c91f22195383bc61348cfba25d4b5b35f9bb7d5ef89 not found: ID does not exist"
Jul 15 12:07:03.498796 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.498714    2565 scope.go:117] "RemoveContainer" containerID="6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"
Jul 15 12:07:03.498971 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:03.498954    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9\": container with ID starting with 6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9 not found: ID does not exist" containerID="6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"
Jul 15 12:07:03.499026 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:03.498982    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9"} err="failed to get container status \"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9\": rpc error: code = NotFound desc = could not find container \"6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9\": container with ID starting with 6f43c96d6458ca021c2a36452d597a60aa8b1b64a02fe1f3dd10c99e01ecbfb9 not found: ID does not exist"
Jul 15 12:07:04.672744 ip-10-0-137-92 kubenswrapper[2565]: E0715 12:07:04.672696    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xk7tb\" is forbidden: User \"system:node:ip-10-0-137-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zx6sn\": no relationship found between node 'ip-10-0-137-92.ec2.internal' and this object" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" pod="openshift-must-gather-zx6sn/must-gather-xk7tb"
Jul 15 12:07:04.674465 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:04.674443    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" path="/var/lib/kubelet/pods/4964f966-0dee-4929-9ccc-cc3361a9535e/volumes"
Jul 15 12:07:04.706900 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:04.706866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:07:04.795247 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:04.795215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q4lmv_732a12cd-410c-4422-9936-8fe6958a978b/konnectivity-agent/0.log" limit=1024
Jul 15 12:07:05.804172 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:05.804139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-45x42_2a1022a7-a38b-422a-a575-7d634cb767a8/manager/0.log"
Jul 15 12:07:05.834985 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:05.834960    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-gc6m6_4d35bccc-aa92-4731-9f68-4ece273ee422/manager/0.log"
Jul 15 12:07:06.304227 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:06.304189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-r4hbr_b7de55b9-da42-4bc9-be97-975ea925f74e/manager/0.log"
Jul 15 12:07:07.868163 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:07.868133    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dpnhr_7f3a4ff0-1cd9-489a-9c26-1b68750d76bd/kube-state-metrics/0.log"
Jul 15 12:07:07.898569 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:07.898539    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dpnhr_7f3a4ff0-1cd9-489a-9c26-1b68750d76bd/kube-rbac-proxy-main/0.log"
Jul 15 12:07:07.927091 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:07.927065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dpnhr_7f3a4ff0-1cd9-489a-9c26-1b68750d76bd/kube-rbac-proxy-self/0.log"
Jul 15 12:07:07.960928 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:07.960906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85777c75db-qjnrg_8a61223d-ad4a-4dde-8886-863969e53b20/metrics-server/0.log"
Jul 15 12:07:07.992313 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:07.992292    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-fmh5p_37d80fc6-8e87-4891-ba49-e6c0d345be2d/monitoring-plugin/0.log"
Jul 15 12:07:08.034198 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.034180    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nr7q_ce9cab10-b113-42f4-9bf7-461420db4623/node-exporter/0.log"
Jul 15 12:07:08.081970 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.081950    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nr7q_ce9cab10-b113-42f4-9bf7-461420db4623/kube-rbac-proxy/0.log"
Jul 15 12:07:08.123397 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.123330    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nr7q_ce9cab10-b113-42f4-9bf7-461420db4623/init-textfile/0.log"
Jul 15 12:07:08.735139 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.735061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kpvpd_2bd74670-f40a-4e73-b8f5-9ace70d532ef/prometheus-operator/0.log"
Jul 15 12:07:08.821720 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.821682    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kpvpd_2bd74670-f40a-4e73-b8f5-9ace70d532ef/kube-rbac-proxy/0.log"
Jul 15 12:07:08.870462 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:08.870434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-564nt_820df008-db97-4b72-b553-4dc34bd8fa3d/prometheus-operator-admission-webhook/0.log"
Jul 15 12:07:09.131056 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.131021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/thanos-query/0.log"
Jul 15 12:07:09.196636 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.196604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/kube-rbac-proxy-web/0.log"
Jul 15 12:07:09.233090 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.233064    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/kube-rbac-proxy/0.log"
Jul 15 12:07:09.268483 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.268447    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/prom-label-proxy/0.log"
Jul 15 12:07:09.297804 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.297762    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/kube-rbac-proxy-rules/0.log"
Jul 15 12:07:09.323482 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:09.323452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fcf7bdd76-5gcrx_a06bd7d9-11bd-4acb-b690-913112365ae7/kube-rbac-proxy-metrics/0.log"
Jul 15 12:07:10.908352 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908319    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"]
Jul 15 12:07:10.908738 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="copy"
Jul 15 12:07:10.908738 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908688    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="copy"
Jul 15 12:07:10.908738 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908715    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="gather"
Jul 15 12:07:10.908738 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="gather"
Jul 15 12:07:10.908935 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908776    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="gather"
Jul 15 12:07:10.908935 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.908787    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4964f966-0dee-4929-9ccc-cc3361a9535e" containerName="copy"
Jul 15 12:07:10.912695 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.912676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:10.915076 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.915037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt44n\"/\"openshift-service-ca.crt\""
Jul 15 12:07:10.915076 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.915068    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt44n\"/\"kube-root-ca.crt\""
Jul 15 12:07:10.915634 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.915609    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lt44n\"/\"default-dockercfg-rg9nd\""
Jul 15 12:07:10.923063 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.923018    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"]
Jul 15 12:07:10.946776 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.946739    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/1.log"
Jul 15 12:07:10.951685 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.951662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-hc5rp_acba8a5e-eb80-4b88-aa63-a33817400d0b/console-operator/2.log"
Jul 15 12:07:10.976587 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.976561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br6d\" (UniqueName: \"kubernetes.io/projected/d60c182c-cb4c-4de4-a62a-4334db4ee644-kube-api-access-6br6d\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:10.976701 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.976611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-podres\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:10.976701 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.976658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-sys\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:10.976701 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.976684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-lib-modules\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:10.976822 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:10.976735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-proc\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077387 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-podres\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-sys\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-lib-modules\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-proc\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br6d\" (UniqueName: \"kubernetes.io/projected/d60c182c-cb4c-4de4-a62a-4334db4ee644-kube-api-access-6br6d\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-sys\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-podres\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-lib-modules\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.077574 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.077566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d60c182c-cb4c-4de4-a62a-4334db4ee644-proc\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.085679 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.085651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br6d\" (UniqueName: \"kubernetes.io/projected/d60c182c-cb4c-4de4-a62a-4334db4ee644-kube-api-access-6br6d\") pod \"perf-node-gather-daemonset-l5bw5\" (UID: \"d60c182c-cb4c-4de4-a62a-4334db4ee644\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.222594 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.222506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.358322 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.358296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"]
Jul 15 12:07:11.360354 ip-10-0-137-92 kubenswrapper[2565]: W0715 12:07:11.360327    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd60c182c_cb4c_4de4_a62a_4334db4ee644.slice/crio-f54292831feeb0acb205266c6408c3bb9dc2957c8be9a37b64ce401cd4a9c79e WatchSource:0}: Error finding container f54292831feeb0acb205266c6408c3bb9dc2957c8be9a37b64ce401cd4a9c79e: Status 404 returned error can't find the container with id f54292831feeb0acb205266c6408c3bb9dc2957c8be9a37b64ce401cd4a9c79e
Jul 15 12:07:11.526240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.526142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5" event={"ID":"d60c182c-cb4c-4de4-a62a-4334db4ee644","Type":"ContainerStarted","Data":"7ea2bb38b87e838406031292a1b602b9309a5502b01f1a62b951cd4127a3c775"}
Jul 15 12:07:11.526240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.526203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5" event={"ID":"d60c182c-cb4c-4de4-a62a-4334db4ee644","Type":"ContainerStarted","Data":"f54292831feeb0acb205266c6408c3bb9dc2957c8be9a37b64ce401cd4a9c79e"}
Jul 15 12:07:11.526240 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.526231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:11.545718 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:11.545673    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5" podStartSLOduration=1.545657758 podStartE2EDuration="1.545657758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:07:11.543453241 +0000 UTC m=+4897.378197718" watchObservedRunningTime="2026-07-15 12:07:11.545657758 +0000 UTC m=+4897.380402207"
Jul 15 12:07:13.169370 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:13.169340    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv7x9_c21dea71-cff4-4203-95e7-b1e5112fa9d2/dns/0.log"
Jul 15 12:07:13.200559 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:13.200538    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv7x9_c21dea71-cff4-4203-95e7-b1e5112fa9d2/kube-rbac-proxy/0.log"
Jul 15 12:07:13.229863 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:13.229841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ckfng_09c549f7-a074-4a6a-a103-6517d81ce013/dns-node-resolver/0.log"
Jul 15 12:07:13.904225 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:13.904192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2h2kf_5a5645dd-b4fd-4aff-a198-f743157e501a/node-ca/0.log"
Jul 15 12:07:15.725351 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:15.725310    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mzhsm_435a8d23-fa77-414b-be7f-0dea1442a91a/discovery/0.log"
Jul 15 12:07:15.823126 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:15.823096    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4b849546-cmq8n_aea22a48-4abf-4d49-9058-99d9ad240187/router/0.log"
Jul 15 12:07:16.396362 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:16.396278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rztx6_b55bedd1-7fc7-4595-a8fa-7b5801ae43ee/serve-healthcheck-canary/0.log"
Jul 15 12:07:17.067316 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:17.067281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdktl_45a962f4-acc0-4a4a-87cf-f3cd6603ddf8/kube-rbac-proxy/0.log"
Jul 15 12:07:17.096844 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:17.096815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdktl_45a962f4-acc0-4a4a-87cf-f3cd6603ddf8/exporter/0.log"
Jul 15 12:07:17.121229 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:17.121201    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdktl_45a962f4-acc0-4a4a-87cf-f3cd6603ddf8/extractor/0.log"
Jul 15 12:07:17.542459 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:17.542427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-l5bw5"
Jul 15 12:07:20.876672 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:20.876646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-vbjvn_75601234-a086-4aeb-b916-b6e08a48c392/server/0.log"
Jul 15 12:07:21.456650 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:21.456569    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-nfnsm_4a3bd5d1-7c26-47d0-b797-0c3bc39987ed/manager/0.log"
Jul 15 12:07:23.024990 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:23.024957    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mzhsm_435a8d23-fa77-414b-be7f-0dea1442a91a/discovery/0.log"
Jul 15 12:07:23.101362 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:23.101331    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4b849546-cmq8n_aea22a48-4abf-4d49-9058-99d9ad240187/router/0.log"
Jul 15 12:07:24.881121 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:24.881085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-mzhsm_435a8d23-fa77-414b-be7f-0dea1442a91a/discovery/0.log"
Jul 15 12:07:24.978465 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:24.978433    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4b849546-cmq8n_aea22a48-4abf-4d49-9058-99d9ad240187/router/0.log"
Jul 15 12:07:27.190976 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:27.190939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-wwxct_bc11f264-bef2-4fd9-92ba-16f892918e19/kube-storage-version-migrator-operator/1.log"
Jul 15 12:07:27.191671 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:27.191649    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-wwxct_bc11f264-bef2-4fd9-92ba-16f892918e19/kube-storage-version-migrator-operator/0.log"
Jul 15 12:07:28.399015 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.398989    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/kube-multus-additional-cni-plugins/0.log"
Jul 15 12:07:28.435763 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.435732    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/egress-router-binary-copy/0.log"
Jul 15 12:07:28.459515 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.459483    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/cni-plugins/0.log"
Jul 15 12:07:28.482389 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.482364    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/bond-cni-plugin/0.log"
Jul 15 12:07:28.506136 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.506113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/routeoverride-cni/0.log"
Jul 15 12:07:28.530220 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.530192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/whereabouts-cni-bincopy/0.log"
Jul 15 12:07:28.554189 ip-10-0-137-92 kubenswrapper[2565]: I0715 12:07:28.554166    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q95q2_9de6dbd2-8336-4020-8b28-0d4e344e9681/whereabouts-cni/0.log"