Sep 18 19:12:00.072546 ip-10-0-138-2 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:12:00.072559 ip-10-0-138-2 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:12:00.072567 ip-10-0-138-2 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:12:00.072930 ip-10-0-138-2 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:12:10.306148 ip-10-0-138-2 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:12:10.306165 ip-10-0-138-2 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3a1955c7ca74315820a0289a47a271c --
Sep 18 19:13:57.799924 ip-10-0-138-2 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:14:01.085913 ip-10-0-138-2 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:01.085913 ip-10-0-138-2 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:14:01.085913 ip-10-0-138-2 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:01.085913 ip-10-0-138-2 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:14:01.085913 ip-10-0-138-2 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:01.086758 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.085984    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:14:01.090484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090464    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.090484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090483    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090487    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090490    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090493    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090496    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090499    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090501    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090504    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090508    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090512    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090516    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090520    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090523    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090526    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090529    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090531    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090534    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090537    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090539    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.090555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090543    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090546    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090549    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090551    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090554    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090557    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090560    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090571    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090575    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090592    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090595    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090599    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090603    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090606    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090609    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090611    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090614    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090617    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090619    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.091030 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090622    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090624    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090627    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090629    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090632    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090634    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090637    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090640    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090643    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090646    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090648    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090651    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090654    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090656    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090659    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090662    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090665    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090668    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090670    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090673    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.091492 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090675    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090678    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090680    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090683    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090686    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090688    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090692    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090696    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090700    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090703    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090705    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090708    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090711    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090714    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090716    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090719    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090721    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090724    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090726    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090729    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.091993 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090732    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090735    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090738    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090740    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090743    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090745    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090748    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090751    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090754    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090756    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090759    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090761    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090764    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090767    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090769    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090772    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090774    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090777    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090779    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.090782    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.092473 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091473    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091479    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091482    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091485    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091487    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091490    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091493    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091496    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091499    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091501    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091505    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091508    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091510    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091513    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091516    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091518    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091521    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091524    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091526    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091529    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.092979 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091532    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091535    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091538    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091541    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091544    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091546    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091549    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091552    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091554    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091556    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091559    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091561    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091564    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091566    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091569    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091571    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091575    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091591    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091594    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.093481 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091597    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091600    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091602    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091605    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091608    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091610    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091613    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091616    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091619    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091621    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091624    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091626    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091629    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091631    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091634    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091637    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091639    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091642    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091644    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091646    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.093967 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091650    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091652    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091655    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091657    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091662    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091665    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091668    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091671    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091673    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091676    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091679    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091681    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091684    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091687    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091691    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091693    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091696    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091698    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091701    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091703    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.094479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091707    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091710    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091715    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091718    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091721    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091724    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091727    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091730    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091733    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091736    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091738    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091741    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091744    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091748    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091751    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091754    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091756    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091759    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091761    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.094986 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.091764    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092173    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092179    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092183    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092185    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092188    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092193    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092196    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092199    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092202    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092204    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092207    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092210    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092214    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092216    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092219    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092222    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092224    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092227    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.095494 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092230    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092232    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092235    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092239    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092242    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092246    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092249    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092251    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092254    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092257    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092260    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092263    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092265    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092269    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092271    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092274    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092277    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092279    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092282    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092286    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.095985 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092288    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092291    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092293    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092296    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092298    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092301    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092304    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092307    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092309    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092312    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092314    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092318    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092321    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092324    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092327    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092330    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092333    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092336    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092338    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.096479 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092341    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092343    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092346    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092348    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092351    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092353    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092356    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092358    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092363    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092365    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092370    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092372    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092376    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092379    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092382    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092384    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092387    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092390    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092392    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092395    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.096981 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092397    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092400    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092403    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092405    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092408    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092410    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092413    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092415    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092418    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092420    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092423    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092426    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092428    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092431    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092433    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092436    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092439    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092443    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092446    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092449    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.097484 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092452    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.092455    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092530    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092590    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092598    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092603    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092608    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092611    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092615    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092620    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092623    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092626    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092630    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092633    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092637    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092640    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092643    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092646    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092649    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092652    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092657    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092660    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092663    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092666    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:14:01.097983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092670    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092674    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092676    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092680    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092683    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092686    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092689    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092692    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092696    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092699    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092704    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092707    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092710    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092713    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092716    2580 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092719    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092724    2580 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092727    2580 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092730    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092733    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092736    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092740    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092743    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092750    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092753    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:14:01.098559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092756    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092759    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092762    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092764    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092767    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092770    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092773    2580 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092777    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092780    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092783    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092787    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092790    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092793    2580 flags.go:64] FLAG: --help="false"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092796    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092799    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092802    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092805    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092809    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092812    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092815    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092818    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092821    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092824    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:14:01.099189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092827    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092831    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092834    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092837    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092840    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092843    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092846    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092848    2580 flags.go:64] FLAG: --lock-file=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092852    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092855    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092858    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092864    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092867    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092870    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092872    2580 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092875    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092879    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092882    2580 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092884    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092888    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092892    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092895    2580 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092899    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092901    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092904    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:14:01.099817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092908    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092911    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092914    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092917    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092924    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092927    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092931    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092934    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092937    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092942    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092946    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092949    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092952    2580 flags.go:64] FLAG: --port="10250"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092955    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092958    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e1b92ef27cb46abb"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092961    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092964    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092967    2580 flags.go:64] FLAG: --register-node="true"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092970    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092973    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092976    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092979    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092982    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092988    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092992    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:14:01.100461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092995    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.092998    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093001    2580 flags.go:64] FLAG: --runonce="false"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093003    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093006    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093011    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093014    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093018    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093021    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093024    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093028    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093031    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093034    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093037    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093040    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093043    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093046    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093049    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093054    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093057    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093060    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093064    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093067    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093070    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093074    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:14:01.101074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093076    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093080    2580 flags.go:64] FLAG: --v="2"
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093084    2580 flags.go:64] FLAG: --version="false"
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093088    2580 flags.go:64] FLAG: --vmodule=""
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093091    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.093095    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093207    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093211    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093214    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093217    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093219    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093223    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093227    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093232    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093234    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093237    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093240    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093242    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093245    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093248    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093251    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.102099 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093253    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093256    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093259    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093262    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093265    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093267    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093270    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093273    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093276    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093278    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093281    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093284    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093287    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093290    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093292    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093295    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093297    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093300    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093302    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093305    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.102686 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093307    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093310    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093313    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093317    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093321    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093324    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093327    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093329    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093333    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093336    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093340    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093344    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093348    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093351    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093354    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093356    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093359    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093361    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093364    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093366    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.103184 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093369    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093372    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093374    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093377    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093381    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093391    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093395    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093398    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093401    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093403    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093407    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093411    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093415    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093417    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093420    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093426    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093430    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093433    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093435    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.103780 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093438    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093440    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093445    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093449    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093452    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093455    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093457    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093460    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093463    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093465    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093468    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093470    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093473    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093476    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093478    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093481    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093484    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093486    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093490    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.104296 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093492    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093495    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093498    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093500    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093503    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.093506    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.094136    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.102616    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.102634    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.102654    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.102664    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102723    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102729    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102732    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.104769 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102736    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102739    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102743    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102748    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102751    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102754    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102758    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102761    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102764    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102766    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102769    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102772    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102775    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102778    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102780    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102783    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102786    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102789    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102792    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102794    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.105114 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102797    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102800    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102803    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102805    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102808    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102810    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102814    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102817    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102819    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102822    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102825    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102828    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102831    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102833    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102836    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102838    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102841    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102844    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102846    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102849    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.105643 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102853    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102856    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102860    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102863    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102866    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102868    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102871    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102873    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102876    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102879    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102883    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102887    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102890    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102893    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102896    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102899    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102902    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102904    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102908    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.106129 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102911    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102914    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102916    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102923    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102925    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102928    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102931    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102934    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102936    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102939    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102942    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102944    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102947    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102949    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102952    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102955    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102958    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102960    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102963    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.106619 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102966    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102969    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102971    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102975    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102977    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102981    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102983    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102986    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102989    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102991    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102994    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102997    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.102999    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103002    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103005    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103007    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.107122 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103010    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.103016    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103127    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103133    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103136    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103139    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103142    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103145    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103147    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103150    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103153    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103156    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103158    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103161    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103164    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:01.107504 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103167    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103169    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103172    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103174    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103177    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103180    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103183    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103185    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103188    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103191    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103195    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103199    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103202    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103205    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103208    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103210    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103213    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103215    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103218    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:01.107960 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103220    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103223    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103226    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103228    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103231    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103233    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103236    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103239    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103241    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103244    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103248    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103251    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103254    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103257    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103260    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103263    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103266    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103269    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103272    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103275    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:01.108420 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103278    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103280    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103282    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103285    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103287    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103290    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103293    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103295    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103298    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103301    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103304    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103306    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103309    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103311    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103314    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103317    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103319    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103322    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103327    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:01.108924 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103330    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103332    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103335    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103337    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103340    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103342    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103345    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103348    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103351    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103354    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103356    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103359    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103362    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103364    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103367    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103369    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103372    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103374    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103377    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103379    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:01.109409 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103382    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103384    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103387    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103390    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103392    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103396    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:01.103399    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.103405    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.103520    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.105850    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.106672    2580 server.go:1019] "Starting client certificate rotation"
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.106775    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.107385    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:01.109904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.108738    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:01.110248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.109954    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:01.117824 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.117803    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:14:01.119551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.119529    2580 log.go:25] "Validated CRI v1 image API"
Sep 18 19:14:01.120366 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.120347    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:14:01.123126 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.123096    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 da2680f2-aebf-4192-a317-649d439fad43:/dev/nvme0n1p4 fa0ee5f7-b331-4272-8804-9be200e39615:/dev/nvme0n1p3]
Sep 18 19:14:01.123126 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.123120    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:14:01.128068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.127940    2580 manager.go:216] Machine: {Timestamp:2026-09-18 19:14:01.127409029 +0000 UTC m=+1.914689424 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102142 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d83eff3bdb414c5a088bb0b7bb6fd SystemUUID:ec2d83ef-f3bd-b414-c5a0-88bb0b7bb6fd BootID:b3a1955c-7ca7-4315-820a-0289a47a271c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:e2:69:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:e2:69:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:5a:1d:7b:bf:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:14:01.128068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.128062    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:14:01.128194 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.128149    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:14:01.129702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.129677    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:14:01.129839 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.129706    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:14:01.130961 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.130951    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:14:01.130997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.130964    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:14:01.130997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.130972    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:01.130997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.130987    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:01.131079 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.131060    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:14:01.134154 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134142    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:01.134263 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134254    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:14:01.134302 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134295    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:14:01.134331 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134311    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:14:01.134331 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134328    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:14:01.134381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134337    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:14:01.134381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134347    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:14:01.134877 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134863    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:01.134877 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.134888    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:01.141858 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.138380    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:14:01.144793 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.144776    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:14:01.144895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.144807    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:14:01.145829 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145818    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145835    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145841    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145846    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145855    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145860    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:14:01.145872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145873    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145879    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145886    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145892    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145900    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145905    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145929    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:14:01.146027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.145935    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:14:01.149566 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.149549    2580 server.go:1268] "Started kubelet"
Sep 18 19:14:01.149723 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.149651    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:14:01.149799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.149746    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:14:01.149847 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.149808    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:14:01.150473 ip-10-0-138-2 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:14:01.151156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.151023    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:14:01.151349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.151317    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:14:01.151671 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.151649    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:14:01.152306 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.152282    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:14:01.152377 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.152284    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:14:01.154259 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.154236    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:14:01.156754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.156736    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:01.158016 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158002    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:14:01.158091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158010    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:01.158191 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.158120    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.158712 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158692    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:14:01.158712 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158695    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:14:01.158824 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158726    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:14:01.158872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158858    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:14:01.158872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158872    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:14:01.158979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158966    2580 factory.go:55] Registering systemd factory
Sep 18 19:14:01.159041 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.158984    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:14:01.159195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159179    2580 factory.go:153] Registering CRI-O factory
Sep 18 19:14:01.159195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159196    2580 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:14:01.159349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159248    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:14:01.159349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159267    2580 factory.go:103] Registering Raw factory
Sep 18 19:14:01.159349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159281    2580 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:14:01.159750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.159734    2580 manager.go:349] Starting recovery of all containers
Sep 18 19:14:01.160278 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.160255    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:14:01.183268 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.183057    2580 manager.go:354] Recovery completed
Sep 18 19:14:01.185664 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.185630    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:14:01.188266 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.188252    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.190033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190012    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.190086 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190048    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.190086 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190058    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.190568 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190555    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:14:01.190620 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190568    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:14:01.190620 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190572    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.190620 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.190599    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:01.191354 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191336    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.191425 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191365    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.191425 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191376    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.191894 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191881    2580 policy_none.go:49] "None policy: Start"
Sep 18 19:14:01.191961 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191897    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:14:01.191961 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.191909    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:14:01.193365 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.193352    2580 policy_none.go:47] "Start"
Sep 18 19:14:01.201634 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.200714    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-2.ec2.internal.18d6802ef8551217  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-2.ec2.internal,UID:ip-10-0-138-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-2.ec2.internal,},FirstTimestamp:2026-09-18 19:14:01.149526551 +0000 UTC m=+1.936806945,LastTimestamp:2026-09-18 19:14:01.149526551 +0000 UTC m=+1.936806945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-2.ec2.internal,}"
Sep 18 19:14:01.204341 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.204322    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tb2r"
Sep 18 19:14:01.209854 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.209787    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-2.ec2.internal.18d6802efabf2bc3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-2.ec2.internal,UID:ip-10-0-138-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-2.ec2.internal,},FirstTimestamp:2026-09-18 19:14:01.190034371 +0000 UTC m=+1.977314765,LastTimestamp:2026-09-18 19:14:01.190034371 +0000 UTC m=+1.977314765,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-2.ec2.internal,}"
Sep 18 19:14:01.214907 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.214833    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-2.ec2.internal.18d6802efabf71d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-2.ec2.internal,UID:ip-10-0-138-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-2.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-2.ec2.internal,},FirstTimestamp:2026-09-18 19:14:01.190052307 +0000 UTC m=+1.977332702,LastTimestamp:2026-09-18 19:14:01.190052307 +0000 UTC m=+1.977332702,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-2.ec2.internal,}"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.224134    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-2.ec2.internal.18d6802efabf98e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-2.ec2.internal,UID:ip-10-0-138-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-2.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-2.ec2.internal,},FirstTimestamp:2026-09-18 19:14:01.190062304 +0000 UTC m=+1.977342698,LastTimestamp:2026-09-18 19:14:01.190062304 +0000 UTC m=+1.977342698,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-2.ec2.internal,}"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.225720    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tb2r"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228136    2580 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.228193    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228207    2580 server.go:87] "Starting device plugin registration server"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228463    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228484    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228563    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228657    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.228665    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.229611    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:14:01.231383 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.229679    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.286449 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.286417    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:14:01.287565 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.287550    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:14:01.287636 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.287612    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:14:01.287636 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.287633    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:14:01.287726 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.287670    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:14:01.291646 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.291619    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:01.329630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.329602    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.331553 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.331534    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.331665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.331563    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.331665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.331574    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.331665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.331622    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.345614 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.345558    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.345614 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.345595    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-2.ec2.internal\": node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.387281 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.387258    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.388257 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.388237    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"]
Sep 18 19:14:01.388315 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.388305    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.389253 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.389236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.389363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.389268    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.389363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.389282    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.391571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.391554    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.391708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.391692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.391768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.391721    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.392340 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392320    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.392414 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392346    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.392414 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392360    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.392512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392419    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.392512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392445    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.392512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.392460    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.394680 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.394662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.394748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.394692    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.395439 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.395424    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.395501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.395446    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.395501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.395456    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.418491 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.418467    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-2.ec2.internal\" not found" node="ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.423011 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.422996    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-2.ec2.internal\" not found" node="ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.460051 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.460022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.460160 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.460065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.460160 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.460082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cbeccd17f22a204da485376383e5ad9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"8cbeccd17f22a204da485376383e5ad9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.487626 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.487557    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.561189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.561333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.561333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cbeccd17f22a204da485376383e5ad9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"8cbeccd17f22a204da485376383e5ad9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.561333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.561333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8cbeccd17f22a204da485376383e5ad9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"8cbeccd17f22a204da485376383e5ad9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.561333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.561309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1ea3ef28fa0b096df0527bcf96440a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"b1ea3ef28fa0b096df0527bcf96440a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.588279 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.588231    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.689258 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.689165    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.723400 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.723365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.727397 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:01.727375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:01.789489 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.789433    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.890040 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.889998    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:01.990619 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:01.990520    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Sep 18 19:14:02.034917 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.034887    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:02.058665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.058631    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:02.100307 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.100276    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:02.101167 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.101151    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:02.102056 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.102042    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:02.106891 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.106873    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:02.107018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.106997    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:02.107063 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.106997    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:02.107063 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.107000    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:02.107063 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.107040    2580 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://ad06820afe55e4cdaa45c8fa39f92d2a-a9f98b7f542b6f26.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.138.2:54852->52.200.64.45:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:02.135196 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.135167    2580 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:02.158515 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.158493    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:02.162220 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.162194    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:02.162552 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.162529    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lx4lc","kube-system/konnectivity-agent-9sxjj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d","openshift-dns/node-resolver-xk6nb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","openshift-multus/multus-wc6ch","openshift-multus/network-metrics-daemon-dwvrr","openshift-cluster-node-tuning-operator/tuned-sp694","openshift-image-registry/node-ca-q6lcg","openshift-multus/multus-additional-cni-plugins-4vwtl","openshift-network-diagnostics/network-check-target-knt79","openshift-network-operator/iptables-alerter-szcbr","openshift-ovn-kubernetes/ovnkube-node-2pp8z"]
Sep 18 19:14:02.165007 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.164986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.165089 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.165071    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:02.167057 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.167031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.169230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.169212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.171455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.171438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.173451 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.173570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173458    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.173570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:02.173570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:02.173570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tq9fm\""
Sep 18 19:14:02.173757 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2n9bn\""
Sep 18 19:14:02.173757 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:02.173820 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.173801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.175851 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.175835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.175928 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.175909    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:02.178129 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.178110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.180351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.180331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.182558 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.182542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.184716 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.184700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:02.184790 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.184753    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:02.186823 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.186805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.189284 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.189264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.198457 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.198436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.200114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.200096    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:02.200896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.200883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.200942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.200885    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:02.233080 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.233042    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:01 +0000 UTC" deadline="2028-05-09 21:09:19.081444908 +0000 UTC"
Sep 18 19:14:02.233080 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.233074    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14377h55m16.848373666s"
Sep 18 19:14:02.259788 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.259766    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:02.265210 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzxn\" (UniqueName: \"kubernetes.io/projected/0abe42a6-25e0-4573-84ca-2f16520a9382-kube-api-access-2hzxn\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.265321 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-kubernetes\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.265321 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-cnibin\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.265321 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovn-node-metrics-cert\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.265508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-bin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.265508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-conf\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.265508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.265508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:02.265508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-device-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-systemd\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458496de-2e73-49dc-92fa-743221adf3b4-host\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-system-cni-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-ovn\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-socket-dir-parent\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.265768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-etc-kubernetes\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e43a5c59-8d21-4cb8-821e-bfda32975870-host-slash\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw529\" (UniqueName: \"kubernetes.io/projected/3c366423-6d3c-469a-899a-e919e630f67a-kube-api-access-pw529\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-bin\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-socket-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cni-binary-copy\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e43a5c59-8d21-4cb8-821e-bfda32975870-iptables-alerter-script\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.265984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-kubelet-config\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-modprobe-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266033 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-host\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-kubelet\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-log-socket\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrnd\" (UniqueName: \"kubernetes.io/projected/bac49455-8cbe-4d26-bfca-eab97241fc4c-kube-api-access-wsrnd\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cnibin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-netns\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92789\" (UniqueName: \"kubernetes.io/projected/e43a5c59-8d21-4cb8-821e-bfda32975870-kube-api-access-92789\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-dbus\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-systemd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-etc-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-slash\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-netd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-script-lib\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-conf-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-kube-api-access-hnzzb\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-run\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-var-lib-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-registration-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-hostroot\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-daemon-config\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-multus-certs\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-lib-modules\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-sys-fs\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-kubelet\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-etc-tuned\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-netns\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0abe42a6-25e0-4573-84ca-2f16520a9382-hosts-file\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.266863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-config\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-k8s-cni-cncf-io\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0abe42a6-25e0-4573-84ca-2f16520a9382-tmp-dir\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.266998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-var-lib-kubelet\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-node-log\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/addcf4d5-8a05-4680-a602-5af3861d16c7-konnectivity-ca\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-multus\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-sys\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-tmp\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-systemd-units\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-env-overrides\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9gb\" (UniqueName: \"kubernetes.io/projected/551f60db-34be-4d58-aa22-2400ebbce962-kube-api-access-qp9gb\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.267351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cr6\" (UniqueName: \"kubernetes.io/projected/458496de-2e73-49dc-92fa-743221adf3b4-kube-api-access-l7cr6\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spw7z\" (UniqueName: \"kubernetes.io/projected/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-kube-api-access-spw7z\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/addcf4d5-8a05-4680-a602-5af3861d16c7-agent-certs\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-system-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/458496de-2e73-49dc-92fa-743221adf3b4-serviceca\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5nt\" (UniqueName: \"kubernetes.io/projected/82e9fb15-d09f-4e68-8b74-301628e9b37b-kube-api-access-5w5nt\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.267875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.267360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysconfig\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.271559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.270278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8xfg\""
Sep 18 19:14:02.271559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.270650    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.289874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4h5m\""
Sep 18 19:14:02.289874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289856    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.289874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f67ck\""
Sep 18 19:14:02.289874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:02.290268 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289886    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.290268 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.289923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.291188 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:02.291188 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291162    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:02.291593 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291569    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.291732 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:02.291798 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:02.291850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.291900 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.292013 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.291996    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.292077 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.292052    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:02.292077 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.292064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.293366 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.293346    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:02.293467 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.293426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.293467 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.293452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:02.347428 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.347390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8l98r\""
Sep 18 19:14:02.348521 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.348505    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6qc5d\""
Sep 18 19:14:02.351517 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.351496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sjwl2\""
Sep 18 19:14:02.351517 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.351508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r9mzq\""
Sep 18 19:14:02.368017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.367992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/addcf4d5-8a05-4680-a602-5af3861d16c7-agent-certs\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.368082 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-system-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.368082 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/458496de-2e73-49dc-92fa-743221adf3b4-serviceca\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.368082 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5nt\" (UniqueName: \"kubernetes.io/projected/82e9fb15-d09f-4e68-8b74-301628e9b37b-kube-api-access-5w5nt\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysconfig\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzxn\" (UniqueName: \"kubernetes.io/projected/0abe42a6-25e0-4573-84ca-2f16520a9382-kube-api-access-2hzxn\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-system-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-kubernetes\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-cnibin\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysconfig\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovn-node-metrics-cert\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-bin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-conf\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-device-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-systemd\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458496de-2e73-49dc-92fa-743221adf3b4-host\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-system-cni-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-cnibin\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368506    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-bin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-kubernetes\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.368602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/458496de-2e73-49dc-92fa-743221adf3b4-serviceca\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-systemd\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-conf\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-system-cni-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-device-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458496de-2e73-49dc-92fa-743221adf3b4-host\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-ovn\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-ovn\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-socket-dir-parent\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-etc-kubernetes\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e43a5c59-8d21-4cb8-821e-bfda32975870-host-slash\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw529\" (UniqueName: \"kubernetes.io/projected/3c366423-6d3c-469a-899a-e919e630f67a-kube-api-access-pw529\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-socket-dir-parent\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-etc-kubernetes\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.369272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e43a5c59-8d21-4cb8-821e-bfda32975870-host-slash\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-bin\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-bin\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.368986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-socket-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cni-binary-copy\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e43a5c59-8d21-4cb8-821e-bfda32975870-iptables-alerter-script\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-kubelet-config\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-modprobe-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-host\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-kubelet\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-log-socket\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-kubelet-config\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrnd\" (UniqueName: \"kubernetes.io/projected/bac49455-8cbe-4d26-bfca-eab97241fc4c-kube-api-access-wsrnd\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cnibin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-modprobe-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-netns\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-host\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-socket-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92789\" (UniqueName: \"kubernetes.io/projected/e43a5c59-8d21-4cb8-821e-bfda32975870-kube-api-access-92789\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-dbus\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cnibin\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-systemd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-etc-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3313fdc-9479-4f30-8880-121063d20652-dbus\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-slash\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-netns\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-netd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-slash\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-log-socket\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-script-lib\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.370923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-conf-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-kube-api-access-hnzzb\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-cni-binary-copy\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.369634    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e43a5c59-8d21-4cb8-821e-bfda32975870-iptables-alerter-script\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-cni-netd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-run\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.369714    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:02.869677177 +0000 UTC m=+3.656957570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-systemd\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-run\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-etc-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-kubelet\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-conf-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-var-lib-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-registration-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-hostroot\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.371708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-var-lib-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-daemon-config\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-registration-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-multus-certs\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-multus-certs\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-lib-modules\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-sys-fs\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-hostroot\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.369974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-lib-modules\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-kubelet\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-cni-dir\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-kubelet\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-etc-tuned\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-netns\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-sys-fs\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.372275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0abe42a6-25e0-4573-84ca-2f16520a9382-hosts-file\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0abe42a6-25e0-4573-84ca-2f16520a9382-hosts-file\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-etc-sysctl-d\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-host-run-netns\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bac49455-8cbe-4d26-bfca-eab97241fc4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-config\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-run-openvswitch\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-multus-daemon-config\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-script-lib\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-k8s-cni-cncf-io\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0abe42a6-25e0-4573-84ca-2f16520a9382-tmp-dir\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.370469    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e9fb15-d09f-4e68-8b74-301628e9b37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-var-lib-kubelet\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.372832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-run-k8s-cni-cncf-io\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.370533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:02.870516227 +0000 UTC m=+3.657796623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-var-lib-kubelet\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-node-log\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/addcf4d5-8a05-4680-a602-5af3861d16c7-konnectivity-ca\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-multus\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-sys\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0abe42a6-25e0-4573-84ca-2f16520a9382-tmp-dir\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-tmp\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-systemd-units\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-env-overrides\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c366423-6d3c-469a-899a-e919e630f67a-sys\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-host-var-lib-cni-multus\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-node-log\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-systemd-units\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.370886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovnkube-config\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e9fb15-d09f-4e68-8b74-301628e9b37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9gb\" (UniqueName: \"kubernetes.io/projected/551f60db-34be-4d58-aa22-2400ebbce962-kube-api-access-qp9gb\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cr6\" (UniqueName: \"kubernetes.io/projected/458496de-2e73-49dc-92fa-743221adf3b4-kube-api-access-l7cr6\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spw7z\" (UniqueName: \"kubernetes.io/projected/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-kube-api-access-spw7z\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-env-overrides\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.371473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/addcf4d5-8a05-4680-a602-5af3861d16c7-konnectivity-ca\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.372424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-ovn-node-metrics-cert\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.372542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/addcf4d5-8a05-4680-a602-5af3861d16c7-agent-certs\") pod \"konnectivity-agent-9sxjj\" (UID: \"addcf4d5-8a05-4680-a602-5af3861d16c7\") " pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.372595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-etc-tuned\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.373861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.373569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c366423-6d3c-469a-899a-e919e630f67a-tmp\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.406878 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.406845    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:02.407031 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.406928    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5bf89"
Sep 18 19:14:02.407941 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.407919    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:02.407941 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.407941    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:02.408083 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.407952    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:02.408083 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.408009    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:02.907993623 +0000 UTC m=+3.695274018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:02.408821 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.408802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/d00bfa2a-33a9-4f95-9c2a-fd47e892db32-kube-api-access-hnzzb\") pod \"multus-wc6ch\" (UID: \"d00bfa2a-33a9-4f95-9c2a-fd47e892db32\") " pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.413720 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.413698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5nt\" (UniqueName: \"kubernetes.io/projected/82e9fb15-d09f-4e68-8b74-301628e9b37b-kube-api-access-5w5nt\") pod \"multus-additional-cni-plugins-4vwtl\" (UID: \"82e9fb15-d09f-4e68-8b74-301628e9b37b\") " pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.417993 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.417967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spw7z\" (UniqueName: \"kubernetes.io/projected/0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5-kube-api-access-spw7z\") pod \"ovnkube-node-2pp8z\" (UID: \"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.417993 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.417978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9gb\" (UniqueName: \"kubernetes.io/projected/551f60db-34be-4d58-aa22-2400ebbce962-kube-api-access-qp9gb\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.424804 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.424780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrnd\" (UniqueName: \"kubernetes.io/projected/bac49455-8cbe-4d26-bfca-eab97241fc4c-kube-api-access-wsrnd\") pod \"aws-ebs-csi-driver-node-csf6d\" (UID: \"bac49455-8cbe-4d26-bfca-eab97241fc4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.427088 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.427065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzxn\" (UniqueName: \"kubernetes.io/projected/0abe42a6-25e0-4573-84ca-2f16520a9382-kube-api-access-2hzxn\") pod \"node-resolver-xk6nb\" (UID: \"0abe42a6-25e0-4573-84ca-2f16520a9382\") " pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.428877 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.428859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw529\" (UniqueName: \"kubernetes.io/projected/3c366423-6d3c-469a-899a-e919e630f67a-kube-api-access-pw529\") pod \"tuned-sp694\" (UID: \"3c366423-6d3c-469a-899a-e919e630f67a\") " pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.444553 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.444521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92789\" (UniqueName: \"kubernetes.io/projected/e43a5c59-8d21-4cb8-821e-bfda32975870-kube-api-access-92789\") pod \"iptables-alerter-szcbr\" (UID: \"e43a5c59-8d21-4cb8-821e-bfda32975870\") " pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.456233 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.456201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cr6\" (UniqueName: \"kubernetes.io/projected/458496de-2e73-49dc-92fa-743221adf3b4-kube-api-access-l7cr6\") pod \"node-ca-q6lcg\" (UID: \"458496de-2e73-49dc-92fa-743221adf3b4\") " pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.470168 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.470116    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ea3ef28fa0b096df0527bcf96440a5.slice/crio-14b17ac99c450f374e5936148bcd267eb24111397afb2a74a47f50b6e7a56874 WatchSource:0}: Error finding container 14b17ac99c450f374e5936148bcd267eb24111397afb2a74a47f50b6e7a56874: Status 404 returned error can't find the container with id 14b17ac99c450f374e5936148bcd267eb24111397afb2a74a47f50b6e7a56874
Sep 18 19:14:02.471215 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.471189    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbeccd17f22a204da485376383e5ad9.slice/crio-bd6cfeb0ed15ed2343e3136d3ab21bb52e96e4636e8c45730fd7d0f51e37df3d WatchSource:0}: Error finding container bd6cfeb0ed15ed2343e3136d3ab21bb52e96e4636e8c45730fd7d0f51e37df3d: Status 404 returned error can't find the container with id bd6cfeb0ed15ed2343e3136d3ab21bb52e96e4636e8c45730fd7d0f51e37df3d
Sep 18 19:14:02.475771 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.475753    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:02.500058 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.500027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:02.509881 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.509853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddcf4d5_8a05_4680_a602_5af3861d16c7.slice/crio-ef5d85303e884ad3452e70a60c83440baca7f84a5fbd838a06830184bc4bb3f3 WatchSource:0}: Error finding container ef5d85303e884ad3452e70a60c83440baca7f84a5fbd838a06830184bc4bb3f3: Status 404 returned error can't find the container with id ef5d85303e884ad3452e70a60c83440baca7f84a5fbd838a06830184bc4bb3f3
Sep 18 19:14:02.521221 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.521195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d"
Sep 18 19:14:02.528424 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.528402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xk6nb"
Sep 18 19:14:02.529735 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.529714    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:02.531836 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.531814    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac49455_8cbe_4d26_bfca_eab97241fc4c.slice/crio-9712e117056d054acb3d7374d602a9c93b6406224dc1a5b2d4b23b88f181b113 WatchSource:0}: Error finding container 9712e117056d054acb3d7374d602a9c93b6406224dc1a5b2d4b23b88f181b113: Status 404 returned error can't find the container with id 9712e117056d054acb3d7374d602a9c93b6406224dc1a5b2d4b23b88f181b113
Sep 18 19:14:02.536549 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.536522    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abe42a6_25e0_4573_84ca_2f16520a9382.slice/crio-c42d36a476240bab4f7c59482d202cbf8088c35edeb86439247e9e4b6c508b31 WatchSource:0}: Error finding container c42d36a476240bab4f7c59482d202cbf8088c35edeb86439247e9e4b6c508b31: Status 404 returned error can't find the container with id c42d36a476240bab4f7c59482d202cbf8088c35edeb86439247e9e4b6c508b31
Sep 18 19:14:02.554938 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.554908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wc6ch"
Sep 18 19:14:02.564243 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.564204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00bfa2a_33a9_4f95_9c2a_fd47e892db32.slice/crio-3586e59dd8f490387766e461ab22b951903558c1e2e640116dce6752116d6095 WatchSource:0}: Error finding container 3586e59dd8f490387766e461ab22b951903558c1e2e640116dce6752116d6095: Status 404 returned error can't find the container with id 3586e59dd8f490387766e461ab22b951903558c1e2e640116dce6752116d6095
Sep 18 19:14:02.568482 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.568460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sp694"
Sep 18 19:14:02.594668 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.594636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6lcg"
Sep 18 19:14:02.604152 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.604125    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458496de_2e73_49dc_92fa_743221adf3b4.slice/crio-3d4637045f27bfb38e06abdbb2e952b4b234941274e20f4bcd8c52e6dee4fb0e WatchSource:0}: Error finding container 3d4637045f27bfb38e06abdbb2e952b4b234941274e20f4bcd8c52e6dee4fb0e: Status 404 returned error can't find the container with id 3d4637045f27bfb38e06abdbb2e952b4b234941274e20f4bcd8c52e6dee4fb0e
Sep 18 19:14:02.607207 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.607188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4vwtl"
Sep 18 19:14:02.615388 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.615360    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e9fb15_d09f_4e68_8b74_301628e9b37b.slice/crio-ea9118371703bf84c38d32d3893bd130aed370d6123140882965428f6cd85bfe WatchSource:0}: Error finding container ea9118371703bf84c38d32d3893bd130aed370d6123140882965428f6cd85bfe: Status 404 returned error can't find the container with id ea9118371703bf84c38d32d3893bd130aed370d6123140882965428f6cd85bfe
Sep 18 19:14:02.625654 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.625630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szcbr"
Sep 18 19:14:02.630806 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.630788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:02.635703 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.635678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43a5c59_8d21_4cb8_821e_bfda32975870.slice/crio-e016fc62c2d21c2c30b14b0c32d3b4f40dcd537e4245827d0ee8b7e5ead8ef9f WatchSource:0}: Error finding container e016fc62c2d21c2c30b14b0c32d3b4f40dcd537e4245827d0ee8b7e5ead8ef9f: Status 404 returned error can't find the container with id e016fc62c2d21c2c30b14b0c32d3b4f40dcd537e4245827d0ee8b7e5ead8ef9f
Sep 18 19:14:02.640029 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:02.640004    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9e5c67_f3b4_4baa_b5a5_ce16acc418d5.slice/crio-aad0d628d5989b449531698d24a65b46c705b691b0f873ca64d6ad1de01d5606 WatchSource:0}: Error finding container aad0d628d5989b449531698d24a65b46c705b691b0f873ca64d6ad1de01d5606: Status 404 returned error can't find the container with id aad0d628d5989b449531698d24a65b46c705b691b0f873ca64d6ad1de01d5606
Sep 18 19:14:02.641901 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.641870    2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-2pp8z_openshift-ovn-kubernetes(0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 18 19:14:02.643170 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.643154    2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-2pp8z_openshift-ovn-kubernetes(0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 18 19:14:02.646310 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.646271    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" podUID="0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5"
Sep 18 19:14:02.876190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.876144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:02.876373 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.876201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:02.876373 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.876341    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.876489 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.876405    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.876385981 +0000 UTC m=+4.663666368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.876489 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.876415    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.876489 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.876482    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.876464267 +0000 UTC m=+4.663744663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.977647 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:02.977608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:02.977860 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.977843    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:02.977923 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.977867    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:02.977923 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.977880    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:02.978056 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:02.977938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.977920408 +0000 UTC m=+4.765200798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.288858 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.288762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:03.289745 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.288901    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:03.289745 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.288960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:03.289745 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.289072    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:03.325066 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.320843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xk6nb" event={"ID":"0abe42a6-25e0-4573-84ca-2f16520a9382","Type":"ContainerStarted","Data":"c42d36a476240bab4f7c59482d202cbf8088c35edeb86439247e9e4b6c508b31"}
Sep 18 19:14:03.325066 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.323924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d" event={"ID":"bac49455-8cbe-4d26-bfca-eab97241fc4c","Type":"ContainerStarted","Data":"9712e117056d054acb3d7374d602a9c93b6406224dc1a5b2d4b23b88f181b113"}
Sep 18 19:14:03.333305 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.333262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"8cbeccd17f22a204da485376383e5ad9","Type":"ContainerStarted","Data":"bd6cfeb0ed15ed2343e3136d3ab21bb52e96e4636e8c45730fd7d0f51e37df3d"}
Sep 18 19:14:03.339461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.339420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"b1ea3ef28fa0b096df0527bcf96440a5","Type":"ContainerStarted","Data":"14b17ac99c450f374e5936148bcd267eb24111397afb2a74a47f50b6e7a56874"}
Sep 18 19:14:03.346229 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.346170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerStarted","Data":"ea9118371703bf84c38d32d3893bd130aed370d6123140882965428f6cd85bfe"}
Sep 18 19:14:03.369165 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.368954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wc6ch" event={"ID":"d00bfa2a-33a9-4f95-9c2a-fd47e892db32","Type":"ContainerStarted","Data":"3586e59dd8f490387766e461ab22b951903558c1e2e640116dce6752116d6095"}
Sep 18 19:14:03.387650 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.387607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sxjj" event={"ID":"addcf4d5-8a05-4680-a602-5af3861d16c7","Type":"ContainerStarted","Data":"ef5d85303e884ad3452e70a60c83440baca7f84a5fbd838a06830184bc4bb3f3"}
Sep 18 19:14:03.406468 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.406430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"aad0d628d5989b449531698d24a65b46c705b691b0f873ca64d6ad1de01d5606"}
Sep 18 19:14:03.412764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.412155    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:02 +0000 UTC" deadline="2028-04-12 19:10:27.672586078 +0000 UTC"
Sep 18 19:14:03.412764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.412219    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13727h56m24.260370592s"
Sep 18 19:14:03.417087 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.416454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szcbr" event={"ID":"e43a5c59-8d21-4cb8-821e-bfda32975870","Type":"ContainerStarted","Data":"e016fc62c2d21c2c30b14b0c32d3b4f40dcd537e4245827d0ee8b7e5ead8ef9f"}
Sep 18 19:14:03.424484 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.422564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6lcg" event={"ID":"458496de-2e73-49dc-92fa-743221adf3b4","Type":"ContainerStarted","Data":"3d4637045f27bfb38e06abdbb2e952b4b234941274e20f4bcd8c52e6dee4fb0e"}
Sep 18 19:14:03.424484 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.424078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3a0df0e827e39567f587ece03a40630a77aaaeacecf070c175b10d086f0631e\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" podUID="0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5"
Sep 18 19:14:03.436693 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.436596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sp694" event={"ID":"3c366423-6d3c-469a-899a-e919e630f67a","Type":"ContainerStarted","Data":"68943aaeb30963e3f6e6efacb1f5c1cbd5e45463a2a486524653b9cbe1feb2cf"}
Sep 18 19:14:03.525570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.525525    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:03.885455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.885414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:03.885678 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.885486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:03.885678 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.885647    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.885794 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.885712    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:05.885693284 +0000 UTC m=+6.672973680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.886162 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.886083    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.886162 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.886136    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:05.886121676 +0000 UTC m=+6.673402074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.986114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:03.986062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:03.986307 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.986235    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:03.986307 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.986256    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:03.986307 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.986271    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.986475 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:03.986328    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:05.986310673 +0000 UTC m=+6.773591061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:04.289170 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:04.289083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:04.289706 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:04.289219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:04.413132 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:04.413083    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:02 +0000 UTC" deadline="2028-02-26 05:33:29.618960715 +0000 UTC"
Sep 18 19:14:04.413132 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:04.413129    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12610h19m25.205836314s"
Sep 18 19:14:04.583671 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:04.583628    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:05.161015 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:05.160982    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:05.290569 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:05.288700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:05.290569 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.288917    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:05.290569 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:05.289322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:05.290569 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.289433    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:05.905293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:05.905253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:05.905516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:05.905317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:05.905516 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.905497    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:05.905516 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.905512    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:05.905731 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.905574    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:09.905553025 +0000 UTC m=+10.692833421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:05.905731 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:05.905608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:09.905597432 +0000 UTC m=+10.692877817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:06.005903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:06.005866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:06.006103 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:06.006081    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:06.006170 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:06.006112    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:06.006170 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:06.006124    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:06.006270 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:06.006185    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:10.006167342 +0000 UTC m=+10.793447743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:06.288191 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:06.288096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:06.288365 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:06.288239    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:07.288999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:07.288962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:07.289451 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:07.288963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:07.289451 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:07.289111    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:07.289451 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:07.289187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:08.287875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:08.287834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:08.288059 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:08.287985    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:09.287907 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:09.287868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:09.287907 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:09.287908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:09.288410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.288064    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:09.288410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.288210    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:09.935972 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:09.935932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:09.936171 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:09.936000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:09.936171 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.936062    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:09.936171 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.936141    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:17.936119083 +0000 UTC m=+18.723399478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:09.936171 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.936140    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:09.936385 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:09.936187    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:17.936175639 +0000 UTC m=+18.723456028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:10.037256 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:10.037219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:10.037459 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:10.037389    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:10.037459 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:10.037415    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:10.037459 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:10.037426    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:10.037633 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:10.037489    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:18.037472302 +0000 UTC m=+18.824752700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:10.288562 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:10.288471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:10.288985 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:10.288632    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:11.289518 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:11.289480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:11.289994 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:11.289620    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:11.289994 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:11.289674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:11.289994 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:11.289823    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:12.288721 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:12.288683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:12.288909 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:12.288803    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:13.288737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:13.288707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:13.289219 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:13.288704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:13.289219 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:13.288837    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:13.289219 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:13.288938    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:14.288312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:14.288269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:14.288499 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:14.288447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:15.287901 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:15.287864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:15.287901 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:15.287893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:15.288490 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:15.288007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:15.288490 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:15.288150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:16.289047 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.288728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:16.289464 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:16.289133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:16.480226 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.480180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6lcg" event={"ID":"458496de-2e73-49dc-92fa-743221adf3b4","Type":"ContainerStarted","Data":"210780f94a653b7c7973aeaabe05f450100ec87b15fda9f23015584ee3e080b4"}
Sep 18 19:14:16.482075 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.482030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sp694" event={"ID":"3c366423-6d3c-469a-899a-e919e630f67a","Type":"ContainerStarted","Data":"734c62ab449a7aad43d914f9ac47286dc221bed2b4d04cda51b92d804b783e76"}
Sep 18 19:14:16.483739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.483712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xk6nb" event={"ID":"0abe42a6-25e0-4573-84ca-2f16520a9382","Type":"ContainerStarted","Data":"ab25946db47d570fefb1793d1cbdb548b95532e0029f1cebedc59518001978bd"}
Sep 18 19:14:16.485346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.485318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d" event={"ID":"bac49455-8cbe-4d26-bfca-eab97241fc4c","Type":"ContainerStarted","Data":"62990c14012bffa07fa33abb507c438060cb67f2e3bcc3ea47c2312b983e5a99"}
Sep 18 19:14:16.486957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.486925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"8cbeccd17f22a204da485376383e5ad9","Type":"ContainerStarted","Data":"c49e46f307f5142cdbbee6c861452631f1ec677627e83c6e28fab3f2825a5992"}
Sep 18 19:14:16.487114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.487084    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:16.488752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.488728    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1ea3ef28fa0b096df0527bcf96440a5" containerID="440a2194f1b7216a13c132251952f6b90925dda9a321ab52101661a6bf1f6cbf" exitCode=0
Sep 18 19:14:16.488843 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.488813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"b1ea3ef28fa0b096df0527bcf96440a5","Type":"ContainerDied","Data":"440a2194f1b7216a13c132251952f6b90925dda9a321ab52101661a6bf1f6cbf"}
Sep 18 19:14:16.490508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.490487    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="35c51e53f4d331e4888e316cec88fefb24e903d107f46b89772843885bb26351" exitCode=0
Sep 18 19:14:16.490606 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.490566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"35c51e53f4d331e4888e316cec88fefb24e903d107f46b89772843885bb26351"}
Sep 18 19:14:16.492230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.492153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wc6ch" event={"ID":"d00bfa2a-33a9-4f95-9c2a-fd47e892db32","Type":"ContainerStarted","Data":"7ed8ff0ba0ee2ae7364215d096f320cb6bbeff510c3e14bb57da1f16b120def0"}
Sep 18 19:14:16.494276 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.494252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sxjj" event={"ID":"addcf4d5-8a05-4680-a602-5af3861d16c7","Type":"ContainerStarted","Data":"5ed5158390c6cd03f81d01e9ff5f07660d1ec4d4944db11824244fc081c8a7c5"}
Sep 18 19:14:16.512533 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.512506    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:16.513419 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.513399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"]
Sep 18 19:14:16.513728 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.513681    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q6lcg" podStartSLOduration=2.731426295 podStartE2EDuration="15.513669391s" totalImagesPullingTime="12.782243096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.511448042 +0000 UTC m=+17.298728445" watchObservedRunningTime="2026-09-18 19:14:16.513669391 +0000 UTC m=+17.300949794"
Sep 18 19:14:16.645760 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.645664    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9sxjj" podStartSLOduration=2.772539619 podStartE2EDuration="15.645639735s" totalImagesPullingTime="12.873100116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.606501015 +0000 UTC m=+17.393781423" watchObservedRunningTime="2026-09-18 19:14:16.645639735 +0000 UTC m=+17.432920138"
Sep 18 19:14:16.645994 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.645962    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wc6ch" podStartSLOduration=2.725382041 podStartE2EDuration="15.645952607s" totalImagesPullingTime="12.920570566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.645504287 +0000 UTC m=+17.432784691" watchObservedRunningTime="2026-09-18 19:14:16.645952607 +0000 UTC m=+17.433233011"
Sep 18 19:14:16.682949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.682822    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sp694" podStartSLOduration=2.831598677 podStartE2EDuration="15.6828001s" totalImagesPullingTime="12.851201423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.682769063 +0000 UTC m=+17.470049468" watchObservedRunningTime="2026-09-18 19:14:16.6828001 +0000 UTC m=+17.470080504"
Sep 18 19:14:16.801708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.801655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xk6nb" podStartSLOduration=2.952964655 podStartE2EDuration="15.801636571s" totalImagesPullingTime="12.848671916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.801282888 +0000 UTC m=+17.588563292" watchObservedRunningTime="2026-09-18 19:14:16.801636571 +0000 UTC m=+17.588916974"
Sep 18 19:14:16.809250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.809218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:16.809997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.809975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:16.820431 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.820406    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:16.915611 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:16.915526    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" podStartSLOduration=0.915505271 podStartE2EDuration="915.505271ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:16.915261295 +0000 UTC m=+17.702541702" watchObservedRunningTime="2026-09-18 19:14:16.915505271 +0000 UTC m=+17.702785674"
Sep 18 19:14:17.235440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.235284    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:16.820429281Z","UUID":"72f78adf-632e-4bf5-8f42-ff8a27f01cb2","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:17.237059 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.237037    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:17.237188 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.237070    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:17.288063 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.288028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:17.288813 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.288745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:17.288987 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.288875    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:17.289424 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.289251    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:17.499141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.499054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szcbr" event={"ID":"e43a5c59-8d21-4cb8-821e-bfda32975870","Type":"ContainerStarted","Data":"c17ed865dcfc118b5f59f0c101b875e4cd0bd656ad8c23cf8a66d143de4a464f"}
Sep 18 19:14:17.501787 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.501743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d" event={"ID":"bac49455-8cbe-4d26-bfca-eab97241fc4c","Type":"ContainerStarted","Data":"f290bc15e359039774f2d53b735519ff6a2a4c932338d7a6458239b89ae001b8"}
Sep 18 19:14:17.504000 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.503960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"b1ea3ef28fa0b096df0527bcf96440a5","Type":"ContainerStarted","Data":"fdc63abb6de23af1ad24585cdb2832d7be41e403342aa7524b01fa15f988db93"}
Sep 18 19:14:17.504238 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.504217    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:17.528128 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.528095    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:17.528324 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.528166    2580 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Sep 18 19:14:17.532390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.532335    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-szcbr" podStartSLOduration=3.781610578 podStartE2EDuration="16.532309853s" totalImagesPullingTime="12.750699275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:17.531760503 +0000 UTC m=+18.319040906" watchObservedRunningTime="2026-09-18 19:14:17.532309853 +0000 UTC m=+18.319590258"
Sep 18 19:14:17.995908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.995686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:17.995908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:17.995756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:17.996146 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.995929    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:17.996146 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.995990    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.995972344 +0000 UTC m=+34.783252729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:17.996146 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.995991    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:17.996146 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:17.996053    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.996036478 +0000 UTC m=+34.783316873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:18.096665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:18.096619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:18.096872 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:18.096824    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:18.096872 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:18.096846    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:18.096872 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:18.096858    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:18.097042 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:18.096937    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.096914894 +0000 UTC m=+34.884195287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:18.288870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:18.288839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:18.289046 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:18.288956    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:18.513436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:18.512939    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:14:18.513436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:18.513017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d" event={"ID":"bac49455-8cbe-4d26-bfca-eab97241fc4c","Type":"ContainerStarted","Data":"da691c0b21aadf14db6af30c65f218acec34861e7e1550ada019e187ffb958c6"}
Sep 18 19:14:18.542692 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:18.541783    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" podStartSLOduration=16.541761353 podStartE2EDuration="16.541761353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:17.573453063 +0000 UTC m=+18.360733468" watchObservedRunningTime="2026-09-18 19:14:18.541761353 +0000 UTC m=+19.329041758"
Sep 18 19:14:19.288344 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:19.288305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:19.288538 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:19.288447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:19.288748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:19.288732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:19.288878 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:19.288836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:20.288852 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:20.288746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:20.289486 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:20.288883    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:21.289207 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:21.289166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:21.289676 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:21.289294    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:21.289676 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:21.289387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:21.289676 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:21.289527    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:22.288054 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:22.288016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:22.288228 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:22.288149    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:22.523733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:22.523699    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="20182ec4b288815ea8c5fe6548c6276e6c622c9f37e016778235f2548ebb3b62" exitCode=0
Sep 18 19:14:22.524129 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:22.523747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"20182ec4b288815ea8c5fe6548c6276e6c622c9f37e016778235f2548ebb3b62"}
Sep 18 19:14:22.565256 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:22.565204    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csf6d" podStartSLOduration=6.931163776 podStartE2EDuration="21.565183301s" totalImagesPullingTime="14.634019525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:18.541679836 +0000 UTC m=+19.328960241" watchObservedRunningTime="2026-09-18 19:14:22.565183301 +0000 UTC m=+23.352463730"
Sep 18 19:14:23.288403 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:23.288370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:23.288403 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:23.288399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:23.288672 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:23.288519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:23.288672 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:23.288658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:23.529050 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:23.528963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerStarted","Data":"62529cbc214de233d641792342286b775604951cd9a0ed7d62f8b67fd22471e1"}
Sep 18 19:14:24.287959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:24.287927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:24.288162 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:24.288037    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:25.288662 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:25.288623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:25.289133 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:25.288673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:25.289133 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:25.288783    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:25.289133 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:25.288922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:26.288779 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:26.288744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:26.289234 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:26.288868    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:27.288597 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.288542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:27.288750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.288615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:27.288750 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:27.288732    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:27.288870 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:27.288835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:27.539771 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.539740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:14:27.540110 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.540089    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5" containerID="5959c086902227a76de4cabb8148aa14ffc16f81c70715c9dab9b379068a041d" exitCode=1
Sep 18 19:14:27.540183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.540162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"97be833cf8a328d6992afeb466802b081e71e8ff259a648560420a580b87f82d"}
Sep 18 19:14:27.540235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.540230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerDied","Data":"5959c086902227a76de4cabb8148aa14ffc16f81c70715c9dab9b379068a041d"}
Sep 18 19:14:27.540282 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.540262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"0b25b037bdfdb25d193887ca02315552079f125c72ce2030ab1f71dd7d40bc5e"}
Sep 18 19:14:27.542091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.542068    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="62529cbc214de233d641792342286b775604951cd9a0ed7d62f8b67fd22471e1" exitCode=0
Sep 18 19:14:27.542213 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:27.542103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"62529cbc214de233d641792342286b775604951cd9a0ed7d62f8b67fd22471e1"}
Sep 18 19:14:28.288380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:28.288346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:28.288564 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:28.288450    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:28.547015 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:28.546991    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:14:28.547663 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:28.547363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"d7a6c21b6ec2554d74197c564a7a750b8a4022a68a7bfc8efd2cd45b1146725b"}
Sep 18 19:14:28.547663 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:28.547429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"e601ad655714df35d39abf03ba59920fad1ff4b1111292f90cbf4ac950775f53"}
Sep 18 19:14:28.547663 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:28.547461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"c6c8128fb56c9cfe0754edff616f70659ac95138e4d13236187d6ba65d5c62cb"}
Sep 18 19:14:29.288136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.288090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:29.288136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.288134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:29.288393 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:29.288230    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:29.288393 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:29.288328    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:29.552392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.552357    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="64299b4d3113c92ab235bf9faabf220a91233b39153404f0ba62f368a2531430" exitCode=0
Sep 18 19:14:29.552880 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.552402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"64299b4d3113c92ab235bf9faabf220a91233b39153404f0ba62f368a2531430"}
Sep 18 19:14:29.710267 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.710228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:29.837198 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:29.837153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:30.139697 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.139606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:30.139849 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.139746    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:14:30.140177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.140158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sxjj"
Sep 18 19:14:30.160310 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.160272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:30.288284 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.288251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:30.288464 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:30.288357    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:30.422330 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.422239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:30.558004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.557971    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:14:30.558461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.558335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"bf300a023cb82a042576d31272dff6e6004b27e39d85f40602dfc348e882bea0"}
Sep 18 19:14:30.673199 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.673104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:30.943120 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:30.943031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.113927 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.113879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.242963 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.242869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.289594 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.289540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:31.289766 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.289628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:31.289813 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:31.289768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:31.289919 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:31.289895    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:31.576905 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.576853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.727641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:31.727603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:32.289128 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.288787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:32.289342 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:32.289151    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:32.381716 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.381657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:32.568256 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.568229    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:14:32.568711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.568684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"66070710e6464a5a08ebcbad66bc4fd7d30e144c8c11e8aae4b94e8636518232"}
Sep 18 19:14:32.569074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.569039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:32.569186 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.569124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:32.569186 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.569173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:32.569319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.569193    2580 scope.go:117] "RemoveContainer" containerID="5959c086902227a76de4cabb8148aa14ffc16f81c70715c9dab9b379068a041d"
Sep 18 19:14:32.584410 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.584365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:32.586499 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.586470    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:32.586695 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:32.586674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:14:33.064977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:33.064932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:33.217228 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:33.217191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:33.288795 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:33.288756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:33.288959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:33.288756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:33.288959 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:33.288898    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:33.289060 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:33.288997    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:34.009068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.009025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:34.009626 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.009093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:34.009626 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.009180    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.009626 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.009202    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.009626 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.009265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret podName:b3313fdc-9479-4f30-8880-121063d20652 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.009241848 +0000 UTC m=+66.796522260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret") pod "global-pull-secret-syncer-lx4lc" (UID: "b3313fdc-9479-4f30-8880-121063d20652") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.009626 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.009302    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.009286186 +0000 UTC m=+66.796566574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.109860 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.109826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:34.109991 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.109956    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:34.109991 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.109973    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:34.109991 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.109982    2580 projected.go:196] Error preparing data for projected volume kube-api-access-77gdx for pod openshift-network-diagnostics/network-check-target-knt79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.110099 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.110035    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx podName:e6051936-2de4-4e0f-a09a-f5b40635578c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.110021279 +0000 UTC m=+66.897301677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-77gdx" (UniqueName: "kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx") pod "network-check-target-knt79" (UID: "e6051936-2de4-4e0f-a09a-f5b40635578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.288086 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.287990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:34.288236 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.288109    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:34.567783 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.567603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lx4lc"]
Sep 18 19:14:34.568004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.567859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:34.568004 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.567969    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:34.576431 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.576404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knt79"]
Sep 18 19:14:34.577054 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.577022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerStarted","Data":"3e9411737b34a32b8888a3b12054bc3803f0246b5fc6c638c2e109a6a6f3384f"}
Sep 18 19:14:34.581444 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.581417    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:14:34.581822 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.581794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" event={"ID":"0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5","Type":"ContainerStarted","Data":"3a2b14d53b7ca2ecc1bd35944ead810d1c17ebd6799510b84201e4db5fa3b114"}
Sep 18 19:14:34.581822 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.581817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:34.581983 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.581928    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:34.585630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.585607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwvrr"]
Sep 18 19:14:34.585728 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.585709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:34.585844 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:34.585820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:34.692597 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:34.692520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z" podStartSLOduration=9.192627839 podStartE2EDuration="33.692501157s" totalImagesPullingTime="24.499873318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-18 19:14:34.689173303 +0000 UTC m=+35.476453708" watchObservedRunningTime="2026-09-18 19:14:34.692501157 +0000 UTC m=+35.479781562"
Sep 18 19:14:35.585882 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:35.585846    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="3e9411737b34a32b8888a3b12054bc3803f0246b5fc6c638c2e109a6a6f3384f" exitCode=0
Sep 18 19:14:35.586346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:35.585922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"3e9411737b34a32b8888a3b12054bc3803f0246b5fc6c638c2e109a6a6f3384f"}
Sep 18 19:14:36.288319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:36.288284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:36.288319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:36.288303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:36.288319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:36.288298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:36.288627 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:36.288442    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:36.288627 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:36.288469    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:36.288627 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:36.288533    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:36.592488 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:36.592456    2580 generic.go:356] "Generic (PLEG): container finished" podID="82e9fb15-d09f-4e68-8b74-301628e9b37b" containerID="21caed5c3eaa10fe8414b78642d938c72802a8bcc9eb80a4391792ebd7b75dc5" exitCode=0
Sep 18 19:14:36.592488 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:36.592494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerDied","Data":"21caed5c3eaa10fe8414b78642d938c72802a8bcc9eb80a4391792ebd7b75dc5"}
Sep 18 19:14:37.597793 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:37.597757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" event={"ID":"82e9fb15-d09f-4e68-8b74-301628e9b37b","Type":"ContainerStarted","Data":"fa2a00cb8e6438efc71c5e523ce64efe848fd8012105b205ae594f51dd2bdc89"}
Sep 18 19:14:37.649871 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:37.649222    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4vwtl" podStartSLOduration=12.426780848 podStartE2EDuration="36.649198972s" totalImagesPullingTime="24.222418124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:37.647152155 +0000 UTC m=+38.434432559" watchObservedRunningTime="2026-09-18 19:14:37.649198972 +0000 UTC m=+38.436479378"
Sep 18 19:14:37.864876 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:37.864786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:38.089023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:38.088988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:38.240343 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:38.240243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:38.288483 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:38.288455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:38.288483 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:38.288480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:38.288673 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:38.288455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:38.288673 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:38.288571    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvrr" podUID="551f60db-34be-4d58-aa22-2400ebbce962"
Sep 18 19:14:38.288673 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:38.288645    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knt79" podUID="e6051936-2de4-4e0f-a09a-f5b40635578c"
Sep 18 19:14:38.288825 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:38.288700    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lx4lc" podUID="b3313fdc-9479-4f30-8880-121063d20652"
Sep 18 19:14:40.149300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.149274    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeReady"
Sep 18 19:14:40.149779 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.149422    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:14:40.286742 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.286650    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h7nph"]
Sep 18 19:14:40.308644 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.308611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l926h"]
Sep 18 19:14:40.308817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.308711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.318693 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.318658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:14:40.318870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.318845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:14:40.318967 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.318876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:14:40.327254 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.327228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:14:40.327392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.327285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.327392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.327315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:14:40.327476 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.327429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:14:40.337987 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.337962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7nph"]
Sep 18 19:14:40.358424 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.358394    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:14:40.358624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.358438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:40.358624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.358443    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:14:40.358624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.358509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:40.361869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.361851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:40.364631 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.364616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:40.364729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.364714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:14:40.366375 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.366355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:14:40.366517 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.366429    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:14:40.366517 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.366432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4mxkp\""
Sep 18 19:14:40.381985 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.381958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l926h"]
Sep 18 19:14:40.453388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58327ff-f218-4763-8ddc-9367f2b79ee8-cert\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.453388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/619098d1-6642-4cb5-814d-236b8dfeee2f-config-volume\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.453705 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dx6\" (UniqueName: \"kubernetes.io/projected/e58327ff-f218-4763-8ddc-9367f2b79ee8-kube-api-access-42dx6\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.453705 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619098d1-6642-4cb5-814d-236b8dfeee2f-tmp-dir\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.453705 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.453705 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.453669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6jz\" (UniqueName: \"kubernetes.io/projected/619098d1-6642-4cb5-814d-236b8dfeee2f-kube-api-access-hc6jz\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554155 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dx6\" (UniqueName: \"kubernetes.io/projected/e58327ff-f218-4763-8ddc-9367f2b79ee8-kube-api-access-42dx6\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.554319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619098d1-6642-4cb5-814d-236b8dfeee2f-tmp-dir\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6jz\" (UniqueName: \"kubernetes.io/projected/619098d1-6642-4cb5-814d-236b8dfeee2f-kube-api-access-hc6jz\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58327ff-f218-4763-8ddc-9367f2b79ee8-cert\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.554319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/619098d1-6642-4cb5-814d-236b8dfeee2f-config-volume\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554573 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:40.554384    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:40.554573 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:40.554465    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.054444239 +0000 UTC m=+41.841724635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:40.554712 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/619098d1-6642-4cb5-814d-236b8dfeee2f-tmp-dir\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.554928 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.554907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/619098d1-6642-4cb5-814d-236b8dfeee2f-config-volume\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.558360 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.558338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58327ff-f218-4763-8ddc-9367f2b79ee8-cert\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.578640 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.578606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dx6\" (UniqueName: \"kubernetes.io/projected/e58327ff-f218-4763-8ddc-9367f2b79ee8-kube-api-access-42dx6\") pod \"ingress-canary-l926h\" (UID: \"e58327ff-f218-4763-8ddc-9367f2b79ee8\") " pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.580675 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.580651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6jz\" (UniqueName: \"kubernetes.io/projected/619098d1-6642-4cb5-814d-236b8dfeee2f-kube-api-access-hc6jz\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:40.642329 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.642291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l926h"
Sep 18 19:14:40.996379 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:40.996061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l926h"]
Sep 18 19:14:41.000555 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:14:41.000525    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58327ff_f218_4763_8ddc_9367f2b79ee8.slice/crio-b00f94a2e74510d02caa00552f570f6ec9006b6ed6d4945fc327278964b824e4 WatchSource:0}: Error finding container b00f94a2e74510d02caa00552f570f6ec9006b6ed6d4945fc327278964b824e4: Status 404 returned error can't find the container with id b00f94a2e74510d02caa00552f570f6ec9006b6ed6d4945fc327278964b824e4
Sep 18 19:14:41.057604 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:41.057546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:41.057781 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:41.057708    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.057818 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:41.057793    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:42.057776424 +0000 UTC m=+42.845056832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.610250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:41.610210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l926h" event={"ID":"e58327ff-f218-4763-8ddc-9367f2b79ee8","Type":"ContainerStarted","Data":"b00f94a2e74510d02caa00552f570f6ec9006b6ed6d4945fc327278964b824e4"}
Sep 18 19:14:42.067388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:42.067353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:42.067572 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:42.067544    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:42.067659 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:42.067639    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:44.067618189 +0000 UTC m=+44.854898576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:43.617514 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:43.617477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l926h" event={"ID":"e58327ff-f218-4763-8ddc-9367f2b79ee8","Type":"ContainerStarted","Data":"a108f6023c42938acf232272c4b795cc571d73f2de485b8d032b463cee6c6fcd"}
Sep 18 19:14:43.649349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:43.649296    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l926h" podStartSLOduration=1.426552327 podStartE2EDuration="3.649276878s" totalImagesPullingTime="2.222724551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:43.648908221 +0000 UTC m=+44.436188626" watchObservedRunningTime="2026-09-18 19:14:43.649276878 +0000 UTC m=+44.436557263"
Sep 18 19:14:44.082805 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:44.082769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:44.082916 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:44.082903    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:44.082964 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:44.082955    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:48.082942124 +0000 UTC m=+48.870222511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:48.114847 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:48.114607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:48.115298 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:48.114775    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:48.115298 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:48.114968    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:14:56.114951202 +0000 UTC m=+56.902231589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:48.142740 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:48.142714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:51.365303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:51.365254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:51.633517 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:51.633432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:51.824805 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:51.824769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:51.967131 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:51.967050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:52.131888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:52.131853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:52.385869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:52.385830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:52.669906 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:52.669813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:53.109028 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:53.108991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:53.223928 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:53.223886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:56.165940 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:14:56.165901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:14:56.166395 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:56.166035    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:56.166395 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:14:56.166107    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:12.166089674 +0000 UTC m=+72.953370058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:00.781965 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:00.781922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:03.862867 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:03.862832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:04.016497 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:04.016462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:04.264875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:04.264774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:05.596412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:05.596383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pp8z"
Sep 18 19:15:05.859083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:05.858998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:06.014398 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.014361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:06.037442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.037407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:15:06.037550 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.037451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:15:06.045777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.045757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:06.046250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.046234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:06.048339 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:06.048323    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:15:06.048395 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:06.048382    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs podName:551f60db-34be-4d58-aa22-2400ebbce962 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:10.048364386 +0000 UTC m=+130.835644768 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs") pod "network-metrics-daemon-dwvrr" (UID: "551f60db-34be-4d58-aa22-2400ebbce962") : secret "metrics-daemon-secret" not found
Sep 18 19:15:06.052144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.052120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3313fdc-9479-4f30-8880-121063d20652-original-pull-secret\") pod \"global-pull-secret-syncer-lx4lc\" (UID: \"b3313fdc-9479-4f30-8880-121063d20652\") " pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:15:06.138832 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.138740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:15:06.142828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.142805    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:06.147895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.147868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lx4lc"
Sep 18 19:15:06.155246 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.155220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:06.161674 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.161643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gdx\" (UniqueName: \"kubernetes.io/projected/e6051936-2de4-4e0f-a09a-f5b40635578c-kube-api-access-77gdx\") pod \"network-check-target-knt79\" (UID: \"e6051936-2de4-4e0f-a09a-f5b40635578c\") " pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:15:06.427364 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.427283    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lx4lc"]
Sep 18 19:15:06.430862 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:06.430834    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3313fdc_9479_4f30_8880_121063d20652.slice/crio-e803dca9fa230f0fa3502f23309c1c570a46ad529ab553a6e112b4e32fec078b WatchSource:0}: Error finding container e803dca9fa230f0fa3502f23309c1c570a46ad529ab553a6e112b4e32fec078b: Status 404 returned error can't find the container with id e803dca9fa230f0fa3502f23309c1c570a46ad529ab553a6e112b4e32fec078b
Sep 18 19:15:06.457037 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.457007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:15:06.463881 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.463844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:15:06.672526 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.672492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lx4lc" event={"ID":"b3313fdc-9479-4f30-8880-121063d20652","Type":"ContainerStarted","Data":"e803dca9fa230f0fa3502f23309c1c570a46ad529ab553a6e112b4e32fec078b"}
Sep 18 19:15:06.758002 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:06.757974    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6051936_2de4_4e0f_a09a_f5b40635578c.slice/crio-47e5d1581dddfeb90ab348708d400c23bbbcda59f2ca2796097a4ff738702aee WatchSource:0}: Error finding container 47e5d1581dddfeb90ab348708d400c23bbbcda59f2ca2796097a4ff738702aee: Status 404 returned error can't find the container with id 47e5d1581dddfeb90ab348708d400c23bbbcda59f2ca2796097a4ff738702aee
Sep 18 19:15:06.758145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:06.758125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knt79"]
Sep 18 19:15:07.473388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:07.473341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:07.563342 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:07.563272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:07.644003 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:07.643960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:07.677439 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:07.677399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knt79" event={"ID":"e6051936-2de4-4e0f-a09a-f5b40635578c","Type":"ContainerStarted","Data":"47e5d1581dddfeb90ab348708d400c23bbbcda59f2ca2796097a4ff738702aee"}
Sep 18 19:15:08.330074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:08.329993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:08.415843 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:08.415809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:11.689930 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:11.689892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lx4lc" event={"ID":"b3313fdc-9479-4f30-8880-121063d20652","Type":"ContainerStarted","Data":"6a85e7268fc4f7c30cdb876c00b8f84376871f2ac12e7fe4d48217491e733d6e"}
Sep 18 19:15:11.691508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:11.691483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knt79" event={"ID":"e6051936-2de4-4e0f-a09a-f5b40635578c","Type":"ContainerStarted","Data":"19de989586e4f6625e268b9765e2fd62101368d4b51ec847d63cacb049acd18e"}
Sep 18 19:15:11.691624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:11.691533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:15:11.709044 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:11.708987    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lx4lc" podStartSLOduration=66.150978984 podStartE2EDuration="1m10.708970947s" totalImagesPullingTime="4.557991963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:11.7087511 +0000 UTC m=+72.496031503" watchObservedRunningTime="2026-09-18 19:15:11.708970947 +0000 UTC m=+72.496251352"
Sep 18 19:15:11.728183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:11.728126    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-knt79" podStartSLOduration=66.501426357 podStartE2EDuration="1m10.728111899s" totalImagesPullingTime="4.226685542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:11.727860759 +0000 UTC m=+72.515141162" watchObservedRunningTime="2026-09-18 19:15:11.728111899 +0000 UTC m=+72.515392302"
Sep 18 19:15:12.184642 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:12.184572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:15:12.184808 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:12.184687    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:12.184808 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:12.184755    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls podName:619098d1-6642-4cb5-814d-236b8dfeee2f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.184736134 +0000 UTC m=+104.972016516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls") pod "dns-default-h7nph" (UID: "619098d1-6642-4cb5-814d-236b8dfeee2f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:16.404688 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.404653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"]
Sep 18 19:15:16.409544 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.409527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.411177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.411153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e871430e-31d9-4f13-8462-e45a0b76452d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.411295 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.411184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.411295 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.411231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvfv\" (UniqueName: \"kubernetes.io/projected/e871430e-31d9-4f13-8462-e45a0b76452d-kube-api-access-ksvfv\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.419761 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.419743    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:15:16.424017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.423986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:15:16.424098 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.423986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:15:16.424098 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.424002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:15:16.424098 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.424032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-prctc\""
Sep 18 19:15:16.430936 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.430914    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7887dff69b-grctd"]
Sep 18 19:15:16.434184 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.434167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.438874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.438842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"]
Sep 18 19:15:16.438979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.438894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 19:15:16.439722 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.439704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p97nc\""
Sep 18 19:15:16.440446 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.440426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 19:15:16.441317 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.441293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 19:15:16.441396 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.441362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 19:15:16.441691 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.441666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:15:16.442250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.442231    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:15:16.474816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.474780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7887dff69b-grctd"]
Sep 18 19:15:16.512024 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.511987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvfv\" (UniqueName: \"kubernetes.io/projected/e871430e-31d9-4f13-8462-e45a0b76452d-kube-api-access-ksvfv\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.512024 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59q6\" (UniqueName: \"kubernetes.io/projected/d2d172c8-0c5e-488e-bf1b-00946e7f692d-kube-api-access-b59q6\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-default-certificate\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e871430e-31d9-4f13-8462-e45a0b76452d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.512288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-stats-auth\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.512517 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:16.512354    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:16.512517 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:16.512412    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:17.012398132 +0000 UTC m=+77.799678535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:16.512886 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.512868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e871430e-31d9-4f13-8462-e45a0b76452d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.553177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.553143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvfv\" (UniqueName: \"kubernetes.io/projected/e871430e-31d9-4f13-8462-e45a0b76452d-kube-api-access-ksvfv\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:16.613166 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.613123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-default-certificate\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.613339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.613204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-stats-auth\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.613339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.613254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.613339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.613277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59q6\" (UniqueName: \"kubernetes.io/projected/d2d172c8-0c5e-488e-bf1b-00946e7f692d-kube-api-access-b59q6\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.613507 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:16.613399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:17.113380685 +0000 UTC m=+77.900661071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:16.613507 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.613395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.613507 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:16.613461    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:16.613692 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:16.613517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:17.113499295 +0000 UTC m=+77.900779682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:16.615549 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.615527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-stats-auth\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.615727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.615707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-default-certificate\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.633359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.633330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59q6\" (UniqueName: \"kubernetes.io/projected/d2d172c8-0c5e-488e-bf1b-00946e7f692d-kube-api-access-b59q6\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:16.841641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.841604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:16.952990 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:16.952958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:17.016228 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:17.016184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:17.016398 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:17.016327    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:17.016398 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:17.016388    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:18.016372901 +0000 UTC m=+78.803653287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:17.116626 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:17.116554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:17.116626 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:17.116610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:17.116772 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:17.116686    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:17.116772 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:17.116725    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:18.116707578 +0000 UTC m=+78.903987965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:17.116772 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:17.116746    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:18.116739775 +0000 UTC m=+78.904020157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:18.022541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:18.022493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:18.022942 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:18.022661    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:18.022942 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:18.022728    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.022712823 +0000 UTC m=+80.809993205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:18.122919 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:18.122870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:18.123092 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:18.122982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:18.123092 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:18.123010    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:18.123092 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:18.123083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.123066593 +0000 UTC m=+80.910346999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:18.123196 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:18.123113    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.123101225 +0000 UTC m=+80.910381606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:20.036449 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.036394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:20.036855 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.036514    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:20.036855 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.036569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.036553342 +0000 UTC m=+84.823833729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:20.137537 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.137502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:20.137654 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.137542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:20.137692 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.137656    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:20.137721 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.137691    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.137674837 +0000 UTC m=+84.924955218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:20.137721 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.137716    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.137709163 +0000 UTC m=+84.924989544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:20.270540 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.270510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:20.284119 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.284091    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ndhng"]
Sep 18 19:15:20.288986 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.288942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.295256 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.295235    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2glk\""
Sep 18 19:15:20.296772 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.296754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:15:20.296863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.296756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:15:20.296863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.296757    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:15:20.296863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.296756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:15:20.314738 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.314706    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndhng"]
Sep 18 19:15:20.339422 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.339390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvc6\" (UniqueName: \"kubernetes.io/projected/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-api-access-whvc6\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.339631 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.339428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.339631 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.339456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.339631 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.339608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f031cd97-0d94-4fed-90c6-69befc4a61e8-crio-socket\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.339777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.339642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f031cd97-0d94-4fed-90c6-69befc4a61e8-data-volume\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.353714 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.353686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:20.440624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvc6\" (UniqueName: \"kubernetes.io/projected/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-api-access-whvc6\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f031cd97-0d94-4fed-90c6-69befc4a61e8-crio-socket\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440868 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f031cd97-0d94-4fed-90c6-69befc4a61e8-data-volume\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440868 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.440782    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.440868 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.440829    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.440868 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.440843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f031cd97-0d94-4fed-90c6-69befc4a61e8-crio-socket\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.440868 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.440853    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.940831478 +0000 UTC m=+81.728111887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.441019 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.440870    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:20.940859643 +0000 UTC m=+81.728140039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.441133 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.441117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f031cd97-0d94-4fed-90c6-69befc4a61e8-data-volume\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.452477 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.452439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvc6\" (UniqueName: \"kubernetes.io/projected/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-api-access-whvc6\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.945452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.945401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.945452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:20.945454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:20.945701 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.945548    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.945701 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.945552    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.945701 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.945637    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.945621516 +0000 UTC m=+82.732901921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.945701 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:20.945698    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.945686582 +0000 UTC m=+82.732966967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.953064 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:21.953016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:21.953064 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:21.953061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:21.953512 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:21.953154    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.953512 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:21.953196    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:21.953512 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:21.953207    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:23.953193753 +0000 UTC m=+84.740474149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.953512 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:21.953260    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:23.953246417 +0000 UTC m=+84.740526800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:22.376104 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:22.376077    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xk6nb_0abe42a6-25e0-4573-84ca-2f16520a9382/dns-node-resolver/0.log"
Sep 18 19:15:23.175420 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:23.175393    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6lcg_458496de-2e73-49dc-92fa-743221adf3b4/node-ca/0.log"
Sep 18 19:15:23.776301 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:23.776269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l926h_e58327ff-f218-4763-8ddc-9367f2b79ee8/serve-healthcheck-canary/0.log"
Sep 18 19:15:23.969801 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:23.969748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:23.969801 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:23.969801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:23.970010 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:23.969915    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:23.970010 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:23.969980    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:27.969965074 +0000 UTC m=+88.757245470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:23.970010 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:23.969917    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:23.970115 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:23.970046    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:27.970035329 +0000 UTC m=+88.757315734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:24.070344 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:24.070299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:24.070506 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:24.070450    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:24.070542 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:24.070520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.07050287 +0000 UTC m=+92.857783252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:24.170724 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:24.170681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:24.170724 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:24.170732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:24.170960 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:24.170836    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.170819966 +0000 UTC m=+92.958100348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:24.170960 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:24.170851    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:24.170960 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:24.170903    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.170885295 +0000 UTC m=+92.958165696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:25.064427 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:25.064388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:25.143602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:25.143556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.997927 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:27.997887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:27.997927 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:27.997925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:27.998410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:27.997993    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:27.998410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:27.998053    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.998039424 +0000 UTC m=+96.785319819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:27.998410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:27.998054    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:27.998410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:27.998109    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.998093442 +0000 UTC m=+96.785373825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:30.284939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.284904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.383388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.383353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.523047 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.523013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.523252 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.523232    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"]
Sep 18 19:15:30.527378 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.527363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"
Sep 18 19:15:30.530619 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.530594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:30.530747 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.530727    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:15:30.532179 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.532163    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wm4b5\""
Sep 18 19:15:30.552164 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.552139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"]
Sep 18 19:15:30.615688 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.615650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.619593 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.619563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pgd\" (UniqueName: \"kubernetes.io/projected/9da3c399-84bd-4702-b82e-1ce2154073f0-kube-api-access-42pgd\") pod \"migrator-76794cbd6c-qlffr\" (UID: \"9da3c399-84bd-4702-b82e-1ce2154073f0\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"
Sep 18 19:15:30.720796 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.720753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pgd\" (UniqueName: \"kubernetes.io/projected/9da3c399-84bd-4702-b82e-1ce2154073f0-kube-api-access-42pgd\") pod \"migrator-76794cbd6c-qlffr\" (UID: \"9da3c399-84bd-4702-b82e-1ce2154073f0\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"
Sep 18 19:15:30.723919 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.723883    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.729655 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.729632    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:15:30.732265 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.732242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pgd\" (UniqueName: \"kubernetes.io/projected/9da3c399-84bd-4702-b82e-1ce2154073f0-kube-api-access-42pgd\") pod \"migrator-76794cbd6c-qlffr\" (UID: \"9da3c399-84bd-4702-b82e-1ce2154073f0\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"
Sep 18 19:15:30.732587 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.732563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.739427 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.739402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:15:30.739543 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.739506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:15:30.739543 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.739511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fkdh2\""
Sep 18 19:15:30.741087 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.741070    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:15:30.745864 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.745847    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:15:30.747679 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.747657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:15:30.821815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt4h\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.821815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.821815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.821815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.822071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.822071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.822071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.821997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.822071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.822026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.836089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.836073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"
Sep 18 19:15:30.857264 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.857240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.923216 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923381 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:30.923348    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:30.923381 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:30.923369    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f888f64-r77qj: secret "image-registry-tls" not found
Sep 18 19:15:30.923649 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:30.923419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls podName:2e76f3f8-ab37-4af8-8d97-5406537f444e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:31.423403139 +0000 UTC m=+92.210683548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls") pod "image-registry-7d9f888f64-r77qj" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e") : secret "image-registry-tls" not found
Sep 18 19:15:30.923649 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923649 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923649 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt4h\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.923950 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.923925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.924064 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.924036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.925157 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.925131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.925686 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.925668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.925968 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.925952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.934908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.934890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.950476 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.950456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt4h\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:30.999361 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:30.999326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:31.103045 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:31.102966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:31.108503 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:31.108448    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr"]
Sep 18 19:15:31.110454 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:31.110431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da3c399_84bd_4702_b82e_1ce2154073f0.slice/crio-066ef0cb33582c4a6a8dc7d9fccb715358c3558e9a8e7f096f2732e8f27073dc WatchSource:0}: Error finding container 066ef0cb33582c4a6a8dc7d9fccb715358c3558e9a8e7f096f2732e8f27073dc: Status 404 returned error can't find the container with id 066ef0cb33582c4a6a8dc7d9fccb715358c3558e9a8e7f096f2732e8f27073dc
Sep 18 19:15:31.428767 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:31.428682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:31.429124 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:31.428867    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:31.429124 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:31.428888    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f888f64-r77qj: secret "image-registry-tls" not found
Sep 18 19:15:31.429124 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:31.428952    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls podName:2e76f3f8-ab37-4af8-8d97-5406537f444e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.42893775 +0000 UTC m=+93.216218133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls") pod "image-registry-7d9f888f64-r77qj" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e") : secret "image-registry-tls" not found
Sep 18 19:15:31.742095 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:31.742023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr" event={"ID":"9da3c399-84bd-4702-b82e-1ce2154073f0","Type":"ContainerStarted","Data":"066ef0cb33582c4a6a8dc7d9fccb715358c3558e9a8e7f096f2732e8f27073dc"}
Sep 18 19:15:32.134477 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.134453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:32.134617 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.134599    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:32.134668 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.134659    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls podName:e871430e-31d9-4f13-8462-e45a0b76452d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:48.134644788 +0000 UTC m=+108.921925175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-msmt9" (UID: "e871430e-31d9-4f13-8462-e45a0b76452d") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:15:32.235389 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.235358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:32.235525 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.235394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:32.235525 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.235480    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:15:32.235525 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.235506    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:48.235489101 +0000 UTC m=+109.022769487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : configmap references non-existent config key: service-ca.crt
Sep 18 19:15:32.235718 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.235530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs podName:d2d172c8-0c5e-488e-bf1b-00946e7f692d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:48.235522195 +0000 UTC m=+109.022802577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs") pod "router-default-7887dff69b-grctd" (UID: "d2d172c8-0c5e-488e-bf1b-00946e7f692d") : secret "router-metrics-certs-default" not found
Sep 18 19:15:32.436888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.436805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:32.437251 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.436960    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:32.437251 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.436978    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f888f64-r77qj: secret "image-registry-tls" not found
Sep 18 19:15:32.437251 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:32.437031    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls podName:2e76f3f8-ab37-4af8-8d97-5406537f444e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:34.437016224 +0000 UTC m=+95.224296610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls") pod "image-registry-7d9f888f64-r77qj" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e") : secret "image-registry-tls" not found
Sep 18 19:15:32.654575 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.654539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.732319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.732235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.746612 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.746561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr" event={"ID":"9da3c399-84bd-4702-b82e-1ce2154073f0","Type":"ContainerStarted","Data":"04c63fbed79a40e7b75dc9b9956e5d76bf04f63f32a7ff9c14781861bbb20918"}
Sep 18 19:15:32.746759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.746641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr" event={"ID":"9da3c399-84bd-4702-b82e-1ce2154073f0","Type":"ContainerStarted","Data":"786778d0083eca9898baaf16e93d4e95f110b3ff637ab5482fb3d45dbe953a53"}
Sep 18 19:15:32.767921 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.767861    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qlffr" podStartSLOduration=1.764724412 podStartE2EDuration="2.767841764s" totalImagesPullingTime="1.003117352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:32.767226031 +0000 UTC m=+93.554506440" watchObservedRunningTime="2026-09-18 19:15:32.767841764 +0000 UTC m=+93.555122169"
Sep 18 19:15:32.881328 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.881295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.968625 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:32.968565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:33.411167 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:33.411135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:33.663697 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:33.663622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:33.789191 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:33.789156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:33.915113 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:33.915035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:34.095839 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:34.095807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:34.245285 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:34.245202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:34.452018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:34.451976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:34.452192 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:34.452135    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:34.452192 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:34.452154    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f888f64-r77qj: secret "image-registry-tls" not found
Sep 18 19:15:34.452266 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:34.452211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls podName:2e76f3f8-ab37-4af8-8d97-5406537f444e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:38.45219667 +0000 UTC m=+99.239477055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls") pod "image-registry-7d9f888f64-r77qj" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e") : secret "image-registry-tls" not found
Sep 18 19:15:34.811270 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:34.811240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:34.940304 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:34.940275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.072083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.072000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.156145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.156102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.359136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.359049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.451676 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.451629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.557821 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.557788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.855671 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:35.855642    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.063632 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:36.063596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:36.063632 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:36.063636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:36.063878 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:36.063858    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.063934 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:15:36.063925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls podName:f031cd97-0d94-4fed-90c6-69befc4a61e8 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:52.063910248 +0000 UTC m=+112.851190630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ndhng" (UID: "f031cd97-0d94-4fed-90c6-69befc4a61e8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.064076 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:36.064062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f031cd97-0d94-4fed-90c6-69befc4a61e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:36.903145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:36.903114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.079678 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.079640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.155418 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.155333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.469405 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.469310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.670515 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.670482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.901312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.901282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.982978 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:37.982949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.057451 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.057416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.413413 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.413383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.480661 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.480620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:38.483061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.483028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"image-registry-7d9f888f64-r77qj\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") " pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:38.544284 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.544227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:38.585955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.585928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.736278 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.736197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.831781 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:38.831753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:15:38.837841 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:38.837806    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e76f3f8_ab37_4af8_8d97_5406537f444e.slice/crio-accb39a6bf0fab1239d5d87b015b95fe85ee57673ca0be1cab9c6964d222f37e WatchSource:0}: Error finding container accb39a6bf0fab1239d5d87b015b95fe85ee57673ca0be1cab9c6964d222f37e: Status 404 returned error can't find the container with id accb39a6bf0fab1239d5d87b015b95fe85ee57673ca0be1cab9c6964d222f37e
Sep 18 19:15:39.047251 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.047219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.454223 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.454138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.670536 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.670500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.771664 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.771568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" event={"ID":"2e76f3f8-ab37-4af8-8d97-5406537f444e","Type":"ContainerStarted","Data":"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"}
Sep 18 19:15:39.771664 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.771640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" event={"ID":"2e76f3f8-ab37-4af8-8d97-5406537f444e","Type":"ContainerStarted","Data":"accb39a6bf0fab1239d5d87b015b95fe85ee57673ca0be1cab9c6964d222f37e"}
Sep 18 19:15:39.771846 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.771728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:15:39.800953 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:39.800905    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" podStartSLOduration=9.800889599 podStartE2EDuration="9.800889599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:39.800726619 +0000 UTC m=+100.588007020" watchObservedRunningTime="2026-09-18 19:15:39.800889599 +0000 UTC m=+100.588170003"
Sep 18 19:15:40.041745 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:40.041649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.638123 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:40.638076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.048069 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.048021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.228602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.228548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.294553 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.294523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.460938 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.460858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.668317 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.668282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.851241 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:41.851200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:42.654823 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:42.654784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:42.696613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:42.696567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-knt79"
Sep 18 19:15:43.049404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:43.049365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.494709 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:43.494618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.860963 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:43.860932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:44.226337 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.226240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:15:44.228496 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.228475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/619098d1-6642-4cb5-814d-236b8dfeee2f-metrics-tls\") pod \"dns-default-h7nph\" (UID: \"619098d1-6642-4cb5-814d-236b8dfeee2f\") " pod="openshift-dns/dns-default-h7nph"
Sep 18 19:15:44.257551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.257521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:44.465636 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.465603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:44.521306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.521244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:15:44.527983 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.527965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7nph"
Sep 18 19:15:44.813685 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:44.813655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7nph"]
Sep 18 19:15:44.816893 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:44.816866    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619098d1_6642_4cb5_814d_236b8dfeee2f.slice/crio-627a38221ebff96ce06822e6c22061ba06bc401fee836dcc9f3fe55ee5f7335e WatchSource:0}: Error finding container 627a38221ebff96ce06822e6c22061ba06bc401fee836dcc9f3fe55ee5f7335e: Status 404 returned error can't find the container with id 627a38221ebff96ce06822e6c22061ba06bc401fee836dcc9f3fe55ee5f7335e
Sep 18 19:15:45.074123 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:45.074037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:45.280288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:45.280257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:45.490568 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:45.490476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:45.795872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:45.795841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7nph" event={"ID":"619098d1-6642-4cb5-814d-236b8dfeee2f","Type":"ContainerStarted","Data":"627a38221ebff96ce06822e6c22061ba06bc401fee836dcc9f3fe55ee5f7335e"}
Sep 18 19:15:46.799812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:46.799785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7nph" event={"ID":"619098d1-6642-4cb5-814d-236b8dfeee2f","Type":"ContainerStarted","Data":"90ef35e1b03b8a5a13adf96dd8399af1b88be9ee7055a5a7523a663f2218ae9f"}
Sep 18 19:15:47.803841 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:47.803806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7nph" event={"ID":"619098d1-6642-4cb5-814d-236b8dfeee2f","Type":"ContainerStarted","Data":"881347c4c057442034d76c6f6508feadfbdb4e9cd24147230ff7712bafb8848e"}
Sep 18 19:15:47.804196 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:47.803871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h7nph"
Sep 18 19:15:47.838707 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:47.838656    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h7nph" podStartSLOduration=66.009152644 podStartE2EDuration="1m7.838640674s" totalImagesPullingTime="1.82948803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:47.83840352 +0000 UTC m=+108.625683923" watchObservedRunningTime="2026-09-18 19:15:47.838640674 +0000 UTC m=+108.625921078"
Sep 18 19:15:48.158390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.158296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:48.160693 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.160670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e871430e-31d9-4f13-8462-e45a0b76452d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-msmt9\" (UID: \"e871430e-31d9-4f13-8462-e45a0b76452d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:48.217726 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.217680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"
Sep 18 19:15:48.258852 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.258819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:48.259027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.258864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:48.259443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.259417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d172c8-0c5e-488e-bf1b-00946e7f692d-service-ca-bundle\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:48.264358 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.264325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d172c8-0c5e-488e-bf1b-00946e7f692d-metrics-certs\") pod \"router-default-7887dff69b-grctd\" (UID: \"d2d172c8-0c5e-488e-bf1b-00946e7f692d\") " pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:48.498665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.498572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9"]
Sep 18 19:15:48.501861 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:48.501827    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode871430e_31d9_4f13_8462_e45a0b76452d.slice/crio-aa33a0b4761f14c86726c9c075a77eba6cf6fb4c2af178e2d41cd17cc6962807 WatchSource:0}: Error finding container aa33a0b4761f14c86726c9c075a77eba6cf6fb4c2af178e2d41cd17cc6962807: Status 404 returned error can't find the container with id aa33a0b4761f14c86726c9c075a77eba6cf6fb4c2af178e2d41cd17cc6962807
Sep 18 19:15:48.543943 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.543913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:48.810465 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.810433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9" event={"ID":"e871430e-31d9-4f13-8462-e45a0b76452d","Type":"ContainerStarted","Data":"aa33a0b4761f14c86726c9c075a77eba6cf6fb4c2af178e2d41cd17cc6962807"}
Sep 18 19:15:48.831390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:48.831369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7887dff69b-grctd"]
Sep 18 19:15:48.833210 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:48.833188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d172c8_0c5e_488e_bf1b_00946e7f692d.slice/crio-022d54db2d69ebe9aebb47ee5504c7b7ab16fc90b16814851cebb4030ff853ac WatchSource:0}: Error finding container 022d54db2d69ebe9aebb47ee5504c7b7ab16fc90b16814851cebb4030ff853ac: Status 404 returned error can't find the container with id 022d54db2d69ebe9aebb47ee5504c7b7ab16fc90b16814851cebb4030ff853ac
Sep 18 19:15:49.814497 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:49.814461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7887dff69b-grctd" event={"ID":"d2d172c8-0c5e-488e-bf1b-00946e7f692d","Type":"ContainerStarted","Data":"2d1bb1dad6cc6b103cb9266f73d2cb23fffa3813dcb9642f1ff96f5e7c57a34b"}
Sep 18 19:15:49.814975 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:49.814510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7887dff69b-grctd" event={"ID":"d2d172c8-0c5e-488e-bf1b-00946e7f692d","Type":"ContainerStarted","Data":"022d54db2d69ebe9aebb47ee5504c7b7ab16fc90b16814851cebb4030ff853ac"}
Sep 18 19:15:49.851268 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:49.851217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7887dff69b-grctd" podStartSLOduration=33.85120078 podStartE2EDuration="33.85120078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:49.85063934 +0000 UTC m=+110.637919745" watchObservedRunningTime="2026-09-18 19:15:49.85120078 +0000 UTC m=+110.638481215"
Sep 18 19:15:50.544213 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.544127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:50.546748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.546724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:50.752867 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.752837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:50.819111 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.819072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9" event={"ID":"e871430e-31d9-4f13-8462-e45a0b76452d","Type":"ContainerStarted","Data":"663b0d5c8e3ff93995ea26c5147a5b2de63532ea5beeab49a7507edc7fb59407"}
Sep 18 19:15:50.819560 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.819280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:50.820500 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.820480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7887dff69b-grctd"
Sep 18 19:15:50.854446 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.854393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-msmt9" podStartSLOduration=33.063959204 podStartE2EDuration="34.854377087s" totalImagesPullingTime="1.790417883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:50.853977358 +0000 UTC m=+111.641257762" watchObservedRunningTime="2026-09-18 19:15:50.854377087 +0000 UTC m=+111.641657492"
Sep 18 19:15:50.994015 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:50.993979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.123652 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:51.123559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.241601 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:51.241553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.089505 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.089449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:52.091770 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.091753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f031cd97-0d94-4fed-90c6-69befc4a61e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndhng\" (UID: \"f031cd97-0d94-4fed-90c6-69befc4a61e8\") " pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:52.097748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.097731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndhng"
Sep 18 19:15:52.373526 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.373446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndhng"]
Sep 18 19:15:52.376108 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:52.376084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf031cd97_0d94_4fed_90c6_69befc4a61e8.slice/crio-37b37e68ec3418ccca7cef9a97a6972a4b54204f2b7e9ade1437361dae5b8ee0 WatchSource:0}: Error finding container 37b37e68ec3418ccca7cef9a97a6972a4b54204f2b7e9ade1437361dae5b8ee0: Status 404 returned error can't find the container with id 37b37e68ec3418ccca7cef9a97a6972a4b54204f2b7e9ade1437361dae5b8ee0
Sep 18 19:15:52.826298 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.826266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndhng" event={"ID":"f031cd97-0d94-4fed-90c6-69befc4a61e8","Type":"ContainerStarted","Data":"ba7130f8c135943d34d2ab29c5976fd5c76235ee537e48b6ff9d2dfa20554950"}
Sep 18 19:15:52.826438 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.826322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndhng" event={"ID":"f031cd97-0d94-4fed-90c6-69befc4a61e8","Type":"ContainerStarted","Data":"37b37e68ec3418ccca7cef9a97a6972a4b54204f2b7e9ade1437361dae5b8ee0"}
Sep 18 19:15:52.954218 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.954159    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:15:52.980228 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.980196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"]
Sep 18 19:15:52.982045 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.982029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:52.988125 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.988101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:15:52.994667 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:52.994649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.020616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.020594    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pvq2c\""
Sep 18 19:15:53.035257 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.035236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"]
Sep 18 19:15:53.096548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.096477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40e45865-d562-4582-b91d-0eb394419b1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qx665\" (UID: \"40e45865-d562-4582-b91d-0eb394419b1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:53.107460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.107436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58cccc5b88-8kqgv"]
Sep 18 19:15:53.109363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.109345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.129644 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.129617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cccc5b88-8kqgv"]
Sep 18 19:15:53.161610 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.161565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.197158 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-installation-pull-secrets\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40e45865-d562-4582-b91d-0eb394419b1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qx665\" (UID: \"40e45865-d562-4582-b91d-0eb394419b1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:53.197303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-certificates\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197429 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-image-registry-private-configuration\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197429 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-tls\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197429 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-trusted-ca\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197552 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1160973b-4bf3-4d10-90c5-c733bfadb18f-ca-trust-extracted\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-bound-sa-token\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.197659 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.197612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgp8\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-kube-api-access-pdgp8\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.199640 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.199616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40e45865-d562-4582-b91d-0eb394419b1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qx665\" (UID: \"40e45865-d562-4582-b91d-0eb394419b1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:53.261217 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.261186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.290172 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.290149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:53.298491 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-image-registry-private-configuration\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.298618 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-tls\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.298745 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-trusted-ca\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.298853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1160973b-4bf3-4d10-90c5-c733bfadb18f-ca-trust-extracted\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.298853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-bound-sa-token\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.298853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgp8\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-kube-api-access-pdgp8\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.299020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-installation-pull-secrets\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.299020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.298914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-certificates\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.299664 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.299621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1160973b-4bf3-4d10-90c5-c733bfadb18f-ca-trust-extracted\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.299999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.299868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-trusted-ca\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.300396 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.300372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-certificates\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.301175 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.301155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-image-registry-private-configuration\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.302054 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.302016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-registry-tls\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.302175 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.302148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1160973b-4bf3-4d10-90c5-c733bfadb18f-installation-pull-secrets\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.310722 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.310703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgp8\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-kube-api-access-pdgp8\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.310895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.310875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1160973b-4bf3-4d10-90c5-c733bfadb18f-bound-sa-token\") pod \"image-registry-58cccc5b88-8kqgv\" (UID: \"1160973b-4bf3-4d10-90c5-c733bfadb18f\") " pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.366342 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.366314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.419624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.419597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.569195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.569166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"]
Sep 18 19:15:53.572058 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:53.572032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e45865_d562_4582_b91d_0eb394419b1c.slice/crio-39c430741de8a97b7d0930a0a30fff1980f86e10cbf3cc8ebcba3aac45212134 WatchSource:0}: Error finding container 39c430741de8a97b7d0930a0a30fff1980f86e10cbf3cc8ebcba3aac45212134: Status 404 returned error can't find the container with id 39c430741de8a97b7d0930a0a30fff1980f86e10cbf3cc8ebcba3aac45212134
Sep 18 19:15:53.690287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.690265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cccc5b88-8kqgv"]
Sep 18 19:15:53.692365 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:15:53.692343    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1160973b_4bf3_4d10_90c5_c733bfadb18f.slice/crio-b593656fa806c26c05f211e7b86772c92a520c5453577b705c7c4ca07d7773d1 WatchSource:0}: Error finding container b593656fa806c26c05f211e7b86772c92a520c5453577b705c7c4ca07d7773d1: Status 404 returned error can't find the container with id b593656fa806c26c05f211e7b86772c92a520c5453577b705c7c4ca07d7773d1
Sep 18 19:15:53.831689 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.831652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndhng" event={"ID":"f031cd97-0d94-4fed-90c6-69befc4a61e8","Type":"ContainerStarted","Data":"b20f7d6c57a3337a621f97328883291a3e839ca3835f216a2ae1d3bf42d5951d"}
Sep 18 19:15:53.833228 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.833204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv" event={"ID":"1160973b-4bf3-4d10-90c5-c733bfadb18f","Type":"ContainerStarted","Data":"fccff4bd442e4b7cc66f796aeb95d83155213c45a819c29cb62e791ccc44e672"}
Sep 18 19:15:53.833378 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.833241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv" event={"ID":"1160973b-4bf3-4d10-90c5-c733bfadb18f","Type":"ContainerStarted","Data":"b593656fa806c26c05f211e7b86772c92a520c5453577b705c7c4ca07d7773d1"}
Sep 18 19:15:53.833378 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.833275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:15:53.834437 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.834416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665" event={"ID":"40e45865-d562-4582-b91d-0eb394419b1c","Type":"ContainerStarted","Data":"39c430741de8a97b7d0930a0a30fff1980f86e10cbf3cc8ebcba3aac45212134"}
Sep 18 19:15:53.863190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:53.863145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv" podStartSLOduration=0.863132868 podStartE2EDuration="863.132868ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:53.862629031 +0000 UTC m=+114.649909436" watchObservedRunningTime="2026-09-18 19:15:53.863132868 +0000 UTC m=+114.650413269"
Sep 18 19:15:55.842545 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.842503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665" event={"ID":"40e45865-d562-4582-b91d-0eb394419b1c","Type":"ContainerStarted","Data":"2d6b54acab9335c326d9a34d3a1a33a91c537cae280c80ff918a335704c487ae"}
Sep 18 19:15:55.842992 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.842613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:55.844664 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.844638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndhng" event={"ID":"f031cd97-0d94-4fed-90c6-69befc4a61e8","Type":"ContainerStarted","Data":"643d55043850135d50b1f79f75933f3e15e208918510d6b973f0966411bd715d"}
Sep 18 19:15:55.847264 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.847246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665"
Sep 18 19:15:55.871169 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.871129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qx665" podStartSLOduration=2.494210302 podStartE2EDuration="3.871116412s" totalImagesPullingTime="1.37690611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:55.870992537 +0000 UTC m=+116.658272942" watchObservedRunningTime="2026-09-18 19:15:55.871116412 +0000 UTC m=+116.658396815"
Sep 18 19:15:55.930861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:55.930813    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ndhng" podStartSLOduration=33.482304719 podStartE2EDuration="35.93079701s" totalImagesPullingTime="2.448492291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:15:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:55.903041249 +0000 UTC m=+116.690321653" watchObservedRunningTime="2026-09-18 19:15:55.93079701 +0000 UTC m=+116.718077414"
Sep 18 19:15:56.112409 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:56.112318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:56.361105 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:56.361071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:56.497485 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:56.497385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:57.812509 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:15:57.812481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h7nph"
Sep 18 19:16:00.506480 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:00.506455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.316026 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.315985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.557805 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.557773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.610767 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.610699    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"]
Sep 18 19:16:02.615288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.615270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.618742 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.618719    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:16:02.618859 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.618768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:16:02.618923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.618906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:16:02.619165 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.619149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2tjw9\""
Sep 18 19:16:02.629235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.629213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"]
Sep 18 19:16:02.632711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.632690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"]
Sep 18 19:16:02.635860 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.635841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.638919 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.638897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 19:16:02.639201 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.639181    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:16:02.639812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.639795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 19:16:02.639891 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.639843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xjljg\""
Sep 18 19:16:02.645994 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.645978    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wll88"]
Sep 18 19:16:02.649295 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.649279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.652880 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.652865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:02.652977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.652866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:02.653173 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.653152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:02.653840 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.653821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-45754\""
Sep 18 19:16:02.655392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.655374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"]
Sep 18 19:16:02.679190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.679163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.766962 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.766929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.766969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.767136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.766992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6g6h\" (UniqueName: \"kubernetes.io/projected/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-api-access-q6g6h\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-tls\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-wtmp\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bce68abf-eb0c-4309-b087-34a2b3a391a5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-sys\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgr74\" (UniqueName: \"kubernetes.io/projected/b2f23b3d-db6f-411c-9f4c-604218090fec-kube-api-access-fgr74\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-metrics-client-ca\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-root\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dsk\" (UniqueName: \"kubernetes.io/projected/f8414775-82dc-4249-b301-5ae01f963546-kube-api-access-q2dsk\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-accelerators-collector-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-textfile\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.767563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.767523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f23b3d-db6f-411c-9f4c-604218090fec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.771531 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.771503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.868089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6g6h\" (UniqueName: \"kubernetes.io/projected/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-api-access-q6g6h\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-tls\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-wtmp\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bce68abf-eb0c-4309-b087-34a2b3a391a5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-sys\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgr74\" (UniqueName: \"kubernetes.io/projected/b2f23b3d-db6f-411c-9f4c-604218090fec-kube-api-access-fgr74\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-metrics-client-ca\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-root\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dsk\" (UniqueName: \"kubernetes.io/projected/f8414775-82dc-4249-b301-5ae01f963546-kube-api-access-q2dsk\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-wtmp\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-accelerators-collector-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-root\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-textfile\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8414775-82dc-4249-b301-5ae01f963546-sys\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f23b3d-db6f-411c-9f4c-604218090fec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.868797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.869282 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.869282 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:16:02.868967    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 19:16:02.869429 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.868973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-metrics-client-ca\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.869610 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:16:02.869033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls podName:b2f23b3d-db6f-411c-9f4c-604218090fec nodeName:}" failed. No retries permitted until 2026-09-18 19:16:03.369016663 +0000 UTC m=+124.156297057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-q28qv" (UID: "b2f23b3d-db6f-411c-9f4c-604218090fec") : secret "openshift-state-metrics-tls" not found
Sep 18 19:16:02.869717 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.869227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bce68abf-eb0c-4309-b087-34a2b3a391a5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.869828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.869366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f23b3d-db6f-411c-9f4c-604218090fec-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.869828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.869815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.869977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.869397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-accelerators-collector-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.869977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.869451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-textfile\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.871312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.871291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.871524 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.871508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8414775-82dc-4249-b301-5ae01f963546-node-exporter-tls\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.871622 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.871600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.871819 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.871803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.872048 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.872031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.880310 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.880286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6g6h\" (UniqueName: \"kubernetes.io/projected/bce68abf-eb0c-4309-b087-34a2b3a391a5-kube-api-access-q6g6h\") pod \"kube-state-metrics-84c885658c-kzgfg\" (UID: \"bce68abf-eb0c-4309-b087-34a2b3a391a5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.881510 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.881460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgr74\" (UniqueName: \"kubernetes.io/projected/b2f23b3d-db6f-411c-9f4c-604218090fec-kube-api-access-fgr74\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:02.881623 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.881550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dsk\" (UniqueName: \"kubernetes.io/projected/f8414775-82dc-4249-b301-5ae01f963546-kube-api-access-q2dsk\") pod \"node-exporter-wll88\" (UID: \"f8414775-82dc-4249-b301-5ae01f963546\") " pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.906516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.906484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.944590 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.944560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"
Sep 18 19:16:02.957807 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.957788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wll88"
Sep 18 19:16:02.959715 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:02.959698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:16:02.967884 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:02.967864    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8414775_82dc_4249_b301_5ae01f963546.slice/crio-352dd8a6787819f9f558947fd3a86c6b7ec8387ad8d9a3509ef23b774887ea02 WatchSource:0}: Error finding container 352dd8a6787819f9f558947fd3a86c6b7ec8387ad8d9a3509ef23b774887ea02: Status 404 returned error can't find the container with id 352dd8a6787819f9f558947fd3a86c6b7ec8387ad8d9a3509ef23b774887ea02
Sep 18 19:16:03.015860 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.015829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.111595 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.111543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.212213 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.212143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.230654 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.230635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-kzgfg"]
Sep 18 19:16:03.232882 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:03.232859    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce68abf_eb0c_4309_b087_34a2b3a391a5.slice/crio-21864715fa3d852fc82a30366b213446cfb0e564d4045bfe5fed77537bbd4749 WatchSource:0}: Error finding container 21864715fa3d852fc82a30366b213446cfb0e564d4045bfe5fed77537bbd4749: Status 404 returned error can't find the container with id 21864715fa3d852fc82a30366b213446cfb0e564d4045bfe5fed77537bbd4749
Sep 18 19:16:03.313248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.313214    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.372029 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.371998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:03.374393 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.374370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f23b3d-db6f-411c-9f4c-604218090fec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q28qv\" (UID: \"b2f23b3d-db6f-411c-9f4c-604218090fec\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:03.524302 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.524227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"
Sep 18 19:16:03.584817 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.584629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.682380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.682351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.827128 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.827093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.834176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.834150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv"]
Sep 18 19:16:03.871988 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.871954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wll88" event={"ID":"f8414775-82dc-4249-b301-5ae01f963546","Type":"ContainerStarted","Data":"352dd8a6787819f9f558947fd3a86c6b7ec8387ad8d9a3509ef23b774887ea02"}
Sep 18 19:16:03.873489 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:03.873463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg" event={"ID":"bce68abf-eb0c-4309-b087-34a2b3a391a5","Type":"ContainerStarted","Data":"21864715fa3d852fc82a30366b213446cfb0e564d4045bfe5fed77537bbd4749"}
Sep 18 19:16:04.013656 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:04.013617    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f23b3d_db6f_411c_9f4c_604218090fec.slice/crio-065469d35a29a22a3c0e0b02394dd0b3693608e58ae5351b5dcdd8475f9556db WatchSource:0}: Error finding container 065469d35a29a22a3c0e0b02394dd0b3693608e58ae5351b5dcdd8475f9556db: Status 404 returned error can't find the container with id 065469d35a29a22a3c0e0b02394dd0b3693608e58ae5351b5dcdd8475f9556db
Sep 18 19:16:04.882296 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.882266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg" event={"ID":"bce68abf-eb0c-4309-b087-34a2b3a391a5","Type":"ContainerStarted","Data":"dd1e475ebbe4630a1b2db9bdc41e76d6aef48b4099d29517e28d38c1ccfb9519"}
Sep 18 19:16:04.882717 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.882335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg" event={"ID":"bce68abf-eb0c-4309-b087-34a2b3a391a5","Type":"ContainerStarted","Data":"c12d4c7e3c59c61f4e966e5447d2ba4a7111ecaecc8a0f39b5b182c646dd5307"}
Sep 18 19:16:04.882717 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.882355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg" event={"ID":"bce68abf-eb0c-4309-b087-34a2b3a391a5","Type":"ContainerStarted","Data":"49496d7d3634a9fd67bd7e8444e2bce60fe56c91dab2e4f35618783a1c1ae162"}
Sep 18 19:16:04.884650 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.884619    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8414775-82dc-4249-b301-5ae01f963546" containerID="1a6047b6a670230928fcc29d1e1c723ea12e89f02eaaa03fe107f0e4ad848b80" exitCode=0
Sep 18 19:16:04.884777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.884715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wll88" event={"ID":"f8414775-82dc-4249-b301-5ae01f963546","Type":"ContainerDied","Data":"1a6047b6a670230928fcc29d1e1c723ea12e89f02eaaa03fe107f0e4ad848b80"}
Sep 18 19:16:04.886990 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.886967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv" event={"ID":"b2f23b3d-db6f-411c-9f4c-604218090fec","Type":"ContainerStarted","Data":"67cb5b113bdaed6f4a6954e6d11f49d8285007d74445bb12fad517d11a3083a0"}
Sep 18 19:16:04.887102 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.887032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv" event={"ID":"b2f23b3d-db6f-411c-9f4c-604218090fec","Type":"ContainerStarted","Data":"00fe49d99ea4b87cfd8d554257724cbe05f821e3fad1517bc4fd05221efdfb98"}
Sep 18 19:16:04.887102 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.887055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv" event={"ID":"b2f23b3d-db6f-411c-9f4c-604218090fec","Type":"ContainerStarted","Data":"065469d35a29a22a3c0e0b02394dd0b3693608e58ae5351b5dcdd8475f9556db"}
Sep 18 19:16:04.960411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:04.960366    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-kzgfg" podStartSLOduration=1.536990207 podStartE2EDuration="2.960344702s" totalImagesPullingTime="1.423354495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:04.958622672 +0000 UTC m=+125.745903076" watchObservedRunningTime="2026-09-18 19:16:04.960344702 +0000 UTC m=+125.747625107"
Sep 18 19:16:05.469990 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.469958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.608347 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.608271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.802939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.802905    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85d47b47bb-8skct"]
Sep 18 19:16:05.806428 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.806413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.817635 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.817610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:16:05.817635 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.817616    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:16:05.817774 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.817691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:16:05.817774 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.817718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:16:05.817908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.817896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4i6rs0l5l3bc2\""
Sep 18 19:16:05.825561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.825539    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:16:05.825673 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.825629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l6qm7\""
Sep 18 19:16:05.853324 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.853295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d47b47bb-8skct"]
Sep 18 19:16:05.892628 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.892535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wll88" event={"ID":"f8414775-82dc-4249-b301-5ae01f963546","Type":"ContainerStarted","Data":"a05cde0475a4fbbe69e23d606e996152a5a8a39c5e51e34b69f4a50095a1955b"}
Sep 18 19:16:05.892628 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.892623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wll88" event={"ID":"f8414775-82dc-4249-b301-5ae01f963546","Type":"ContainerStarted","Data":"581d166983a2919c0c0878dfab48bbddb280f8419f6d26c5557c5d3449fc5684"}
Sep 18 19:16:05.894842 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.894809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv" event={"ID":"b2f23b3d-db6f-411c-9f4c-604218090fec","Type":"ContainerStarted","Data":"45af13aea07d72594368f24c55ef4dc892cea46f4c993dd2fe823f1e56553397"}
Sep 18 19:16:05.896337 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84038c7f-0898-42f5-89ef-6025e7f2aa19-metrics-client-ca\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896521 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrkr\" (UniqueName: \"kubernetes.io/projected/84038c7f-0898-42f5-89ef-6025e7f2aa19-kube-api-access-xlrkr\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896574 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-grpc-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.896725 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.896685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.920149 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.920126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.935077 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.935042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q28qv" podStartSLOduration=2.797591828 podStartE2EDuration="3.935029392s" totalImagesPullingTime="1.137437564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:05.934964395 +0000 UTC m=+126.722244800" watchObservedRunningTime="2026-09-18 19:16:05.935029392 +0000 UTC m=+126.722309796"
Sep 18 19:16:05.997040 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.996997    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wll88" podStartSLOduration=2.830230609 podStartE2EDuration="3.996978721s" totalImagesPullingTime="1.090750383s" totalInitContainerRuntime="75.997729ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:05.995912844 +0000 UTC m=+126.783193242" watchObservedRunningTime="2026-09-18 19:16:05.996978721 +0000 UTC m=+126.784259127"
Sep 18 19:16:05.997232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.997296 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84038c7f-0898-42f5-89ef-6025e7f2aa19-metrics-client-ca\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.997346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.997564 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.997666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.997666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.997661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrkr\" (UniqueName: \"kubernetes.io/projected/84038c7f-0898-42f5-89ef-6025e7f2aa19-kube-api-access-xlrkr\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.998043 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.998010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.998952 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.998254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-grpc-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:05.998952 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:05.998715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84038c7f-0898-42f5-89ef-6025e7f2aa19-metrics-client-ca\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.000476 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.000431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.000916 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.000890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.001032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.000898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.001032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.000975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.001032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.000973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.001232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.001209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84038c7f-0898-42f5-89ef-6025e7f2aa19-secret-grpc-tls\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.015749 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.015726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrkr\" (UniqueName: \"kubernetes.io/projected/84038c7f-0898-42f5-89ef-6025e7f2aa19-kube-api-access-xlrkr\") pod \"thanos-querier-85d47b47bb-8skct\" (UID: \"84038c7f-0898-42f5-89ef-6025e7f2aa19\") " pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.034134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.034098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:06.115624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.115574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:06.203683 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.203654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:06.315258 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.315219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:06.405029 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.405008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d47b47bb-8skct"]
Sep 18 19:16:06.407126 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:06.407101    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84038c7f_0898_42f5_89ef_6025e7f2aa19.slice/crio-86b9f7e4146cbab54c71f85cd31b60614c99ed385b9a22c08e758325ebbd6f4f WatchSource:0}: Error finding container 86b9f7e4146cbab54c71f85cd31b60614c99ed385b9a22c08e758325ebbd6f4f: Status 404 returned error can't find the container with id 86b9f7e4146cbab54c71f85cd31b60614c99ed385b9a22c08e758325ebbd6f4f
Sep 18 19:16:06.899444 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:06.899409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"86b9f7e4146cbab54c71f85cd31b60614c99ed385b9a22c08e758325ebbd6f4f"}
Sep 18 19:16:07.153753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.153633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:07.302197 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.302077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:07.380028 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.379990    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"]
Sep 18 19:16:07.383665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.383645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.388851 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.388831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 19:16:07.389032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.389015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dsdbx\""
Sep 18 19:16:07.399230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.399205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"]
Sep 18 19:16:07.430847 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.430782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:07.512827 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.512798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlttb\" (UniqueName: \"kubernetes.io/projected/b002d540-be83-429f-8b5d-7e5698f302d9-kube-api-access-dlttb\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.512988 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.512837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b002d540-be83-429f-8b5d-7e5698f302d9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.535874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.535830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:07.614188 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.614155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b002d540-be83-429f-8b5d-7e5698f302d9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.614356 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.614254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlttb\" (UniqueName: \"kubernetes.io/projected/b002d540-be83-429f-8b5d-7e5698f302d9-kube-api-access-dlttb\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.616530 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.616503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b002d540-be83-429f-8b5d-7e5698f302d9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.623669 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.623650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlttb\" (UniqueName: \"kubernetes.io/projected/b002d540-be83-429f-8b5d-7e5698f302d9-kube-api-access-dlttb\") pod \"monitoring-plugin-5984459ff8-kp5bh\" (UID: \"b002d540-be83-429f-8b5d-7e5698f302d9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.694979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.694907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:07.966232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:07.966014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"]
Sep 18 19:16:08.339251 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:08.339222    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb002d540_be83_429f_8b5d_7e5698f302d9.slice/crio-58071db730e87803380d286a8de79d9bde7470b5917bbb291f080000608e3f47 WatchSource:0}: Error finding container 58071db730e87803380d286a8de79d9bde7470b5917bbb291f080000608e3f47: Status 404 returned error can't find the container with id 58071db730e87803380d286a8de79d9bde7470b5917bbb291f080000608e3f47
Sep 18 19:16:08.909242 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:08.909204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh" event={"ID":"b002d540-be83-429f-8b5d-7e5698f302d9","Type":"ContainerStarted","Data":"58071db730e87803380d286a8de79d9bde7470b5917bbb291f080000608e3f47"}
Sep 18 19:16:08.911718 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:08.911686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"4cfc578a8e079918088824efcbe6c39a02af4b003010ba43add2d8886ed75bae"}
Sep 18 19:16:08.911825 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:08.911751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"7543522744eab5013d9f54d9f6854573ecd6187d7b826796b5c23591b703b479"}
Sep 18 19:16:08.911825 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:08.911789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"7e3958ef311e6b9ca03f0ab3b7168df7dbccb7f099713f76e4f57c1042ccaa32"}
Sep 18 19:16:09.447750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.447714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:09.916911 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.916861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh" event={"ID":"b002d540-be83-429f-8b5d-7e5698f302d9","Type":"ContainerStarted","Data":"534254a617bafa8e80977d9c5f4ad4019feeed130f3ab931a9a9e3a0405484e1"}
Sep 18 19:16:09.917083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.917024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:09.920349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.920320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"d35a875cf0cf3709225a729d857e32be4820a2a6f410e8e2f214d6e3f0b6c995"}
Sep 18 19:16:09.920473 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.920388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"341a6aabcd797aca576649097d12c94ef8bdc019e60bd2ded42e4bf447f26bd4"}
Sep 18 19:16:09.920473 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.920433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" event={"ID":"84038c7f-0898-42f5-89ef-6025e7f2aa19","Type":"ContainerStarted","Data":"f531aee04823a66b3533401542795e2296331b8a709e37ef08236738edf3a3f9"}
Sep 18 19:16:09.920623 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.920475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:09.922019 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.922000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh"
Sep 18 19:16:09.964023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:09.963940    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-kp5bh" podStartSLOduration=1.638999348 podStartE2EDuration="2.963927537s" totalImagesPullingTime="1.324928189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:09.963602019 +0000 UTC m=+130.750882418" watchObservedRunningTime="2026-09-18 19:16:09.963927537 +0000 UTC m=+130.751207942"
Sep 18 19:16:10.005465 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.005416    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct" podStartSLOduration=1.938048585 podStartE2EDuration="5.005401472s" totalImagesPullingTime="3.067352887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:10.002973276 +0000 UTC m=+130.790253680" watchObservedRunningTime="2026-09-18 19:16:10.005401472 +0000 UTC m=+130.792681866"
Sep 18 19:16:10.136221 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.136180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:16:10.138387 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.138357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551f60db-34be-4d58-aa22-2400ebbce962-metrics-certs\") pod \"network-metrics-daemon-dwvrr\" (UID: \"551f60db-34be-4d58-aa22-2400ebbce962\") " pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:16:10.341233 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.341205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:16:10.348013 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.347995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvrr"
Sep 18 19:16:10.636356 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.636332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwvrr"]
Sep 18 19:16:10.638267 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:10.638246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551f60db_34be_4d58_aa22_2400ebbce962.slice/crio-3cc1052811cc99e5c2304a1df6a921d2778990f2217d85b81812cca58d268149 WatchSource:0}: Error finding container 3cc1052811cc99e5c2304a1df6a921d2778990f2217d85b81812cca58d268149: Status 404 returned error can't find the container with id 3cc1052811cc99e5c2304a1df6a921d2778990f2217d85b81812cca58d268149
Sep 18 19:16:10.929706 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:10.929618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvrr" event={"ID":"551f60db-34be-4d58-aa22-2400ebbce962","Type":"ContainerStarted","Data":"3cc1052811cc99e5c2304a1df6a921d2778990f2217d85b81812cca58d268149"}
Sep 18 19:16:11.935458 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:11.935362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvrr" event={"ID":"551f60db-34be-4d58-aa22-2400ebbce962","Type":"ContainerStarted","Data":"c6d499fef08296601e469cd0629423b43b1d675a31289b702ebc7437b9299f60"}
Sep 18 19:16:11.935458 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:11.935417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvrr" event={"ID":"551f60db-34be-4d58-aa22-2400ebbce962","Type":"ContainerStarted","Data":"9fddf39881d31de87b14ee1773cf9e5cb5ec62d255712480bdead3435ac66a05"}
Sep 18 19:16:11.962573 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:11.962527    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dwvrr" podStartSLOduration=129.97854824 podStartE2EDuration="2m10.962513624s" totalImagesPullingTime="983.965384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:11.962111273 +0000 UTC m=+132.749391677" watchObservedRunningTime="2026-09-18 19:16:11.962513624 +0000 UTC m=+132.749794027"
Sep 18 19:16:12.143483 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:12.143458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:12.249453 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:12.249377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:14.529892 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:14.529856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:14.815317 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:14.815290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:14.842388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:14.842368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58cccc5b88-8kqgv"
Sep 18 19:16:15.205748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:15.205669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:15.472450 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:15.472366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:15.936425 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:15.936400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85d47b47bb-8skct"
Sep 18 19:16:17.972191 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:17.972154    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" podUID="2e76f3f8-ab37-4af8-8d97-5406537f444e" containerName="registry" containerID="cri-o://7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676" gracePeriod=30
Sep 18 19:16:18.209089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.209067    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:16:18.301456 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppt4h\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301613 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.301914 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.301656    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token\") pod \"2e76f3f8-ab37-4af8-8d97-5406537f444e\" (UID: \"2e76f3f8-ab37-4af8-8d97-5406537f444e\") "
Sep 18 19:16:18.302046 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.302010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:18.302373 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.302266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:18.304203 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.304174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:18.304300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.304185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:18.304300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.304218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h" (OuterVolumeSpecName: "kube-api-access-ppt4h") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "kube-api-access-ppt4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:18.304501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.304483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:18.304728 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.304701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:18.310854 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.310830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2e76f3f8-ab37-4af8-8d97-5406537f444e" (UID: "2e76f3f8-ab37-4af8-8d97-5406537f444e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:18.402432 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402408    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402432 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402432    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-image-registry-private-configuration\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402441    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e76f3f8-ab37-4af8-8d97-5406537f444e-installation-pull-secrets\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402451    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e76f3f8-ab37-4af8-8d97-5406537f444e-ca-trust-extracted\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402460    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-registry-certificates\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402468    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppt4h\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-kube-api-access-ppt4h\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402476    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e76f3f8-ab37-4af8-8d97-5406537f444e-trusted-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.402616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.402485    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e76f3f8-ab37-4af8-8d97-5406537f444e-bound-sa-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:16:18.961572 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.961537    2580 generic.go:356] "Generic (PLEG): container finished" podID="2e76f3f8-ab37-4af8-8d97-5406537f444e" containerID="7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676" exitCode=0
Sep 18 19:16:18.961776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.961594    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj"
Sep 18 19:16:18.961776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.961625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" event={"ID":"2e76f3f8-ab37-4af8-8d97-5406537f444e","Type":"ContainerDied","Data":"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"}
Sep 18 19:16:18.961776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.961667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f888f64-r77qj" event={"ID":"2e76f3f8-ab37-4af8-8d97-5406537f444e","Type":"ContainerDied","Data":"accb39a6bf0fab1239d5d87b015b95fe85ee57673ca0be1cab9c6964d222f37e"}
Sep 18 19:16:18.961776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.961687    2580 scope.go:117] "RemoveContainer" containerID="7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"
Sep 18 19:16:18.970312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.970296    2580 scope.go:117] "RemoveContainer" containerID="7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"
Sep 18 19:16:18.970635 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:16:18.970611    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676\": container with ID starting with 7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676 not found: ID does not exist" containerID="7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"
Sep 18 19:16:18.970728 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.970641    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676"} err="failed to get container status \"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676\": rpc error: code = NotFound desc = could not find container \"7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676\": container with ID starting with 7cc8a6bed0700939c95665f758829f6f0cada1bec9a49911de288d236a210676 not found: ID does not exist"
Sep 18 19:16:18.991248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:18.989375    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:16:19.001968 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:19.001941    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d9f888f64-r77qj"]
Sep 18 19:16:19.293646 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:19.293546    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e76f3f8-ab37-4af8-8d97-5406537f444e" path="/var/lib/kubelet/pods/2e76f3f8-ab37-4af8-8d97-5406537f444e/volumes"
Sep 18 19:16:20.558923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.558890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.632451 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.632420    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-q5jdx"]
Sep 18 19:16:20.632733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.632721    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e76f3f8-ab37-4af8-8d97-5406537f444e" containerName="registry"
Sep 18 19:16:20.632773 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.632736    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e76f3f8-ab37-4af8-8d97-5406537f444e" containerName="registry"
Sep 18 19:16:20.632818 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.632799    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e76f3f8-ab37-4af8-8d97-5406537f444e" containerName="registry"
Sep 18 19:16:20.636387 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.636371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:20.650992 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.650970    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:16:20.651122 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.651044    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:16:20.651122 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.651108    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6f2kq\""
Sep 18 19:16:20.757313 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.757283    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-q5jdx"]
Sep 18 19:16:20.762828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.762801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.820745 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.820680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqvx\" (UniqueName: \"kubernetes.io/projected/8030ffbd-33ee-45c7-858b-9491d5902930-kube-api-access-sqqvx\") pod \"downloads-6594f4c75-q5jdx\" (UID: \"8030ffbd-33ee-45c7-858b-9491d5902930\") " pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:20.883678 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.883646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.921639 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.921613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqvx\" (UniqueName: \"kubernetes.io/projected/8030ffbd-33ee-45c7-858b-9491d5902930-kube-api-access-sqqvx\") pod \"downloads-6594f4c75-q5jdx\" (UID: \"8030ffbd-33ee-45c7-858b-9491d5902930\") " pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:20.933809 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.933783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqvx\" (UniqueName: \"kubernetes.io/projected/8030ffbd-33ee-45c7-858b-9491d5902930-kube-api-access-sqqvx\") pod \"downloads-6594f4c75-q5jdx\" (UID: \"8030ffbd-33ee-45c7-858b-9491d5902930\") " pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:20.944873 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:20.944854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:21.056632 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:21.056596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:21.227893 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:21.227872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-q5jdx"]
Sep 18 19:16:21.229565 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:21.229539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8030ffbd_33ee_45c7_858b_9491d5902930.slice/crio-68c3e5e8e855187623714f8383ae0d17e54fc44eabe3b8a395c7a9e2da6c5876 WatchSource:0}: Error finding container 68c3e5e8e855187623714f8383ae0d17e54fc44eabe3b8a395c7a9e2da6c5876: Status 404 returned error can't find the container with id 68c3e5e8e855187623714f8383ae0d17e54fc44eabe3b8a395c7a9e2da6c5876
Sep 18 19:16:21.292295 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:21.292266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:21.975431 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:21.975392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-q5jdx" event={"ID":"8030ffbd-33ee-45c7-858b-9491d5902930","Type":"ContainerStarted","Data":"68c3e5e8e855187623714f8383ae0d17e54fc44eabe3b8a395c7a9e2da6c5876"}
Sep 18 19:16:22.334146 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:22.334106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.439089 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:22.439055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.535165 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:22.535119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.698096 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:22.697941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.969016 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:22.968938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.060020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:23.059976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.184551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:23.184515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.259874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:23.259787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:30.357435 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:30.357391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:30.527915 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:30.527876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:30.688412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:30.688333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:30.792698 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:30.792664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:32.176126 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.176096    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:16:32.180516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.180499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.190468 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.190444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:16:32.190638 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.190477    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:16:32.190638 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.190525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:16:32.192180 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.192156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7r2zw\""
Sep 18 19:16:32.192297 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.192199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:16:32.192297 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.192199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:16:32.206260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.206382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.206382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.206511 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9f2n\" (UniqueName: \"kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.206511 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.206511 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.206452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.215556 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.215522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:16:32.306939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.306903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.306939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.306950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.306976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.307026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f2n\" (UniqueName: \"kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.307049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.307085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307785 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.307757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.307904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.307802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.308038 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.308020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.309999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.309979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.310098 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.310007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.317449 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.317420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f2n\" (UniqueName: \"kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n\") pod \"console-55875bcc7d-qhqwz\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") " pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:32.491311 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:32.491199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:36.897771 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:36.897693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:16:36.901011 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:36.900976    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0127dc_a5c3_4715_a1aa_981851b94d17.slice/crio-8c87dbe69302b0ba0f4c929132e406bb9e644703ca94f0641d3bf617dc29c4dd WatchSource:0}: Error finding container 8c87dbe69302b0ba0f4c929132e406bb9e644703ca94f0641d3bf617dc29c4dd: Status 404 returned error can't find the container with id 8c87dbe69302b0ba0f4c929132e406bb9e644703ca94f0641d3bf617dc29c4dd
Sep 18 19:16:37.039707 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.039637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55875bcc7d-qhqwz" event={"ID":"7e0127dc-a5c3-4715-a1aa-981851b94d17","Type":"ContainerStarted","Data":"8c87dbe69302b0ba0f4c929132e406bb9e644703ca94f0641d3bf617dc29c4dd"}
Sep 18 19:16:37.041536 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.041507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-q5jdx" event={"ID":"8030ffbd-33ee-45c7-858b-9491d5902930","Type":"ContainerStarted","Data":"2bc913803d92d7ddb8fd6a72d8c15d8978d278c206f5fa7a92b03d8b21272f3e"}
Sep 18 19:16:37.041810 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.041776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:37.057330 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.057307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-q5jdx"
Sep 18 19:16:37.111433 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.111381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-q5jdx" podStartSLOduration=1.610652998 podStartE2EDuration="17.111364552s" totalImagesPullingTime="15.500711554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:37.07767878 +0000 UTC m=+157.864959186" watchObservedRunningTime="2026-09-18 19:16:37.111364552 +0000 UTC m=+157.898645015"
Sep 18 19:16:37.981602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:37.981544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:38.087393 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:38.087326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:40.762067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:40.762005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:40.910085 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:40.910038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:41.079573 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:41.079537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:42.519794 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.519758    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:16:42.533887 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.533863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.546165 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.546145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:16:42.553478 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.553450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.582548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.582456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkxr\" (UniqueName: \"kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683320 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683320 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683320 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683320 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683320 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkxr\" (UniqueName: \"kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.683599 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.683327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.684287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.684254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.684713 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.684687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.684896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.684874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.684972 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.684901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.687078 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.687055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.687708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.687687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.699664 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.699622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkxr\" (UniqueName: \"kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr\") pod \"console-5c9fc4cb77-nst4t\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") " pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:42.845278 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:42.845241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:43.076787 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:43.076750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55875bcc7d-qhqwz" event={"ID":"7e0127dc-a5c3-4715-a1aa-981851b94d17","Type":"ContainerStarted","Data":"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"}
Sep 18 19:16:43.116487 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:43.116393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55875bcc7d-qhqwz" podStartSLOduration=5.490765614 podStartE2EDuration="11.116375133s" totalImagesPullingTime="5.625609519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:43.113833118 +0000 UTC m=+163.901113521" watchObservedRunningTime="2026-09-18 19:16:43.116375133 +0000 UTC m=+163.903655536"
Sep 18 19:16:43.163077 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:43.163029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:16:43.167797 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:16:43.167770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9313224b_4aa0_4a4f_b580_68c1b3dd44dd.slice/crio-451f0c099b411eed5866708cde549dc308e8afa77ca531d5052e779ed4000e0f WatchSource:0}: Error finding container 451f0c099b411eed5866708cde549dc308e8afa77ca531d5052e779ed4000e0f: Status 404 returned error can't find the container with id 451f0c099b411eed5866708cde549dc308e8afa77ca531d5052e779ed4000e0f
Sep 18 19:16:44.087746 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:44.087700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fc4cb77-nst4t" event={"ID":"9313224b-4aa0-4a4f-b580-68c1b3dd44dd","Type":"ContainerStarted","Data":"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"}
Sep 18 19:16:44.088236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:44.087796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fc4cb77-nst4t" event={"ID":"9313224b-4aa0-4a4f-b580-68c1b3dd44dd","Type":"ContainerStarted","Data":"451f0c099b411eed5866708cde549dc308e8afa77ca531d5052e779ed4000e0f"}
Sep 18 19:16:44.119529 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:44.119434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c9fc4cb77-nst4t" podStartSLOduration=2.119418524 podStartE2EDuration="2.119418524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:44.117777167 +0000 UTC m=+164.905057584" watchObservedRunningTime="2026-09-18 19:16:44.119418524 +0000 UTC m=+164.906698927"
Sep 18 19:16:47.628379 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:47.628349    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-msmt9_e871430e-31d9-4f13-8462-e45a0b76452d/cluster-monitoring-operator/0.log"
Sep 18 19:16:47.824300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:47.824271    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-state-metrics/0.log"
Sep 18 19:16:48.029488 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:48.029406    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-rbac-proxy-main/0.log"
Sep 18 19:16:48.275764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:48.275740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-rbac-proxy-self/0.log"
Sep 18 19:16:48.625931 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:48.625905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-kp5bh_b002d540-be83-429f-8b5d-7e5698f302d9/monitoring-plugin/0.log"
Sep 18 19:16:50.020399 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:50.020369    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/init-textfile/0.log"
Sep 18 19:16:50.240711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:50.240678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/node-exporter/0.log"
Sep 18 19:16:50.422513 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:50.422483    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/kube-rbac-proxy/0.log"
Sep 18 19:16:50.615203 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:50.615171    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/kube-rbac-proxy-main/0.log"
Sep 18 19:16:50.846395 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:50.846366    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/kube-rbac-proxy-self/0.log"
Sep 18 19:16:51.016354 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:51.016317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/openshift-state-metrics/0.log"
Sep 18 19:16:52.491370 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.491335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:52.491781 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.491404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:52.496219 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.496195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:52.845514 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.845480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:52.845702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.845538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:52.850430 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:52.850403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:53.017081 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.017052    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-qx665_40e45865-d562-4582-b91d-0eb394419b1c/prometheus-operator-admission-webhook/0.log"
Sep 18 19:16:53.126017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.125903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:16:53.126017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.125978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:53.126230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.126035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:16:53.220532 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.220499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/thanos-query/0.log"
Sep 18 19:16:53.223436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.223406    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:16:53.232091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.232064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:53.416772 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.416681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-web/0.log"
Sep 18 19:16:53.614443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.614406    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy/0.log"
Sep 18 19:16:53.696949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.696865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:53.814268 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:53.814242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/prom-label-proxy/0.log"
Sep 18 19:16:54.028032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.027946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-rules/0.log"
Sep 18 19:16:54.239797 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.239763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-metrics/0.log"
Sep 18 19:16:54.259731 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.259695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:54.338329 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.338303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:54.500482 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.500445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:54.604386 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:54.604268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:55.015081 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:55.015002    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55875bcc7d-qhqwz_7e0127dc-a5c3-4715-a1aa-981851b94d17/console/0.log"
Sep 18 19:16:55.220452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:55.220424    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c9fc4cb77-nst4t_9313224b-4aa0-4a4f-b580-68c1b3dd44dd/console/0.log"
Sep 18 19:16:55.423204 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:55.423172    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-q5jdx_8030ffbd-33ee-45c7-858b-9491d5902930/download-server/0.log"
Sep 18 19:16:55.502541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:55.502506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:55.666284 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:55.666253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:58.094540 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:16:58.094447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:07.893691 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:07.893656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:07.997813 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:07.997777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:20.148600 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.148543    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55875bcc7d-qhqwz" podUID="7e0127dc-a5c3-4715-a1aa-981851b94d17" containerName="console" containerID="cri-o://700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9" gracePeriod=15
Sep 18 19:17:20.447447 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.447414    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:17:20.508348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9f2n\" (UniqueName: \"kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n\") pod \"7e0127dc-a5c3-4715-a1aa-981851b94d17\" (UID: \"7e0127dc-a5c3-4715-a1aa-981851b94d17\") "
Sep 18 19:17:20.508816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.508789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:20.509029 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.509000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config" (OuterVolumeSpecName: "console-config") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:20.509108 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.509057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:20.510775 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.510741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:20.510887 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.510775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:20.510887 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.510826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n" (OuterVolumeSpecName: "kube-api-access-l9f2n") pod "7e0127dc-a5c3-4715-a1aa-981851b94d17" (UID: "7e0127dc-a5c3-4715-a1aa-981851b94d17"). InnerVolumeSpecName "kube-api-access-l9f2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:20.609519 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609477    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-oauth-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:20.609519 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609512    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-oauth-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:20.609519 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609521    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:20.609519 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609530    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e0127dc-a5c3-4715-a1aa-981851b94d17-console-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:20.609816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609541    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9f2n\" (UniqueName: \"kubernetes.io/projected/7e0127dc-a5c3-4715-a1aa-981851b94d17-kube-api-access-l9f2n\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:20.609816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:20.609550    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e0127dc-a5c3-4715-a1aa-981851b94d17-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:17:21.243470 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.243437    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e0127dc-a5c3-4715-a1aa-981851b94d17" containerID="700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9" exitCode=0
Sep 18 19:17:21.243944 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.243479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55875bcc7d-qhqwz"
Sep 18 19:17:21.243944 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.243526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55875bcc7d-qhqwz" event={"ID":"7e0127dc-a5c3-4715-a1aa-981851b94d17","Type":"ContainerDied","Data":"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"}
Sep 18 19:17:21.243944 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.243651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55875bcc7d-qhqwz" event={"ID":"7e0127dc-a5c3-4715-a1aa-981851b94d17","Type":"ContainerDied","Data":"8c87dbe69302b0ba0f4c929132e406bb9e644703ca94f0641d3bf617dc29c4dd"}
Sep 18 19:17:21.243944 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.243683    2580 scope.go:117] "RemoveContainer" containerID="700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"
Sep 18 19:17:21.252609 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.252570    2580 scope.go:117] "RemoveContainer" containerID="700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"
Sep 18 19:17:21.252923 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:17:21.252902    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9\": container with ID starting with 700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9 not found: ID does not exist" containerID="700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"
Sep 18 19:17:21.252980 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.252933    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9"} err="failed to get container status \"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9\": rpc error: code = NotFound desc = could not find container \"700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9\": container with ID starting with 700bed9f7638da641677dcd4b80f522fe9a43e729d09364c233dab42281ee3f9 not found: ID does not exist"
Sep 18 19:17:21.277336 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.277307    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:17:21.280822 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.280796    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55875bcc7d-qhqwz"]
Sep 18 19:17:21.293263 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:21.293235    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0127dc-a5c3-4715-a1aa-981851b94d17" path="/var/lib/kubelet/pods/7e0127dc-a5c3-4715-a1aa-981851b94d17/volumes"
Sep 18 19:17:22.496185 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:22.496154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:22.966911 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:22.966879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:23.064363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:23.064328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:27.598325 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.598289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-db94bc99d-7kzjv"]
Sep 18 19:17:27.598835 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.598604    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0127dc-a5c3-4715-a1aa-981851b94d17" containerName="console"
Sep 18 19:17:27.598835 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.598615    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0127dc-a5c3-4715-a1aa-981851b94d17" containerName="console"
Sep 18 19:17:27.598835 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.598668    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0127dc-a5c3-4715-a1aa-981851b94d17" containerName="console"
Sep 18 19:17:27.601776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.601753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.606156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.605901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 19:17:27.606156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.605965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 19:17:27.606156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.606003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 19:17:27.606156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.606079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 19:17:27.606156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.606141    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 19:17:27.606673 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.606238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9hh5q\""
Sep 18 19:17:27.612328 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.612308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 19:17:27.619404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.619379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db94bc99d-7kzjv"]
Sep 18 19:17:27.771739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8k5\" (UniqueName: \"kubernetes.io/projected/0534aae7-245b-4984-8f24-1a8ce2fa89a4-kube-api-access-kl8k5\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-serving-certs-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-federate-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-metrics-client-ca\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.771955 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.772107 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.771971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8k5\" (UniqueName: \"kubernetes.io/projected/0534aae7-245b-4984-8f24-1a8ce2fa89a4-kube-api-access-kl8k5\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-serving-certs-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-federate-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-metrics-client-ca\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.873777 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.873615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.874294 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.874266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-metrics-client-ca\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.874404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.874266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-serving-certs-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.874467 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.874450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.876151 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.876126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.876379 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.876359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.876379 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.876373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-telemeter-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.876663 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.876646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0534aae7-245b-4984-8f24-1a8ce2fa89a4-federate-client-tls\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.888612 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.888571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8k5\" (UniqueName: \"kubernetes.io/projected/0534aae7-245b-4984-8f24-1a8ce2fa89a4-kube-api-access-kl8k5\") pod \"telemeter-client-db94bc99d-7kzjv\" (UID: \"0534aae7-245b-4984-8f24-1a8ce2fa89a4\") " pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:27.913433 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:27.913407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv"
Sep 18 19:17:28.198283 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:28.198259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db94bc99d-7kzjv"]
Sep 18 19:17:28.200078 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:17:28.200051    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0534aae7_245b_4984_8f24_1a8ce2fa89a4.slice/crio-75c97555d7dd38164e33c545517da5b9efdce7a3d5529d66a3955ab5bbafa5ac WatchSource:0}: Error finding container 75c97555d7dd38164e33c545517da5b9efdce7a3d5529d66a3955ab5bbafa5ac: Status 404 returned error can't find the container with id 75c97555d7dd38164e33c545517da5b9efdce7a3d5529d66a3955ab5bbafa5ac
Sep 18 19:17:28.273693 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:28.273655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv" event={"ID":"0534aae7-245b-4984-8f24-1a8ce2fa89a4","Type":"ContainerStarted","Data":"75c97555d7dd38164e33c545517da5b9efdce7a3d5529d66a3955ab5bbafa5ac"}
Sep 18 19:17:30.283560 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:30.283468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv" event={"ID":"0534aae7-245b-4984-8f24-1a8ce2fa89a4","Type":"ContainerStarted","Data":"0bd31ab688fd32e6b59e10101ac3894a3fddfc19b81f12f473abbdb374a03f19"}
Sep 18 19:17:31.292949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.292863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv" event={"ID":"0534aae7-245b-4984-8f24-1a8ce2fa89a4","Type":"ContainerStarted","Data":"de02d8d16e5aa39682e0f68e94392cfd6106d5c8abbbb66e048ee75a47855c99"}
Sep 18 19:17:31.292949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.292916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv" event={"ID":"0534aae7-245b-4984-8f24-1a8ce2fa89a4","Type":"ContainerStarted","Data":"1bc13e4961cec5560fe1f01f2ad7283f7a78d785f72ed3f5aa2fd051d1e65ac3"}
Sep 18 19:17:31.332061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.332001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-db94bc99d-7kzjv" podStartSLOduration=1.662707717 podStartE2EDuration="4.331985s" totalImagesPullingTime="2.669277283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:31.331306822 +0000 UTC m=+212.118587227" watchObservedRunningTime="2026-09-18 19:17:31.331985 +0000 UTC m=+212.119265404"
Sep 18 19:17:31.467040 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.467003    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:17:31.469244 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.469221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.484399 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.484371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:17:31.541355 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.541324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:31.614975 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.614943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmg55\" (UniqueName: \"kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.614998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.615055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.615098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.615139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.615170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.615306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.615254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.721364 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.721323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.721548 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.721380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmg55\" (UniqueName: \"kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.721632 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.721567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722000 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.721980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.722895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.723035 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.723035 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.722968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.724316 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.724294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.724467 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.724448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.732783 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.732757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmg55\" (UniqueName: \"kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55\") pod \"console-cd46f7b7-jl6rk\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") " pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:31.778711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:31.778675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:32.056497 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:32.056465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:17:32.059992 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:17:32.059959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bc8edb_16db_4c3b_b7f6_75d0556e2d50.slice/crio-851c2f3248c150b46bd5430d9982b3c1b4fee6d1038cfca2cf4856df3797bdd9 WatchSource:0}: Error finding container 851c2f3248c150b46bd5430d9982b3c1b4fee6d1038cfca2cf4856df3797bdd9: Status 404 returned error can't find the container with id 851c2f3248c150b46bd5430d9982b3c1b4fee6d1038cfca2cf4856df3797bdd9
Sep 18 19:17:32.295824 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:32.295786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd46f7b7-jl6rk" event={"ID":"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50","Type":"ContainerStarted","Data":"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"}
Sep 18 19:17:32.296224 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:32.295842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd46f7b7-jl6rk" event={"ID":"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50","Type":"ContainerStarted","Data":"851c2f3248c150b46bd5430d9982b3c1b4fee6d1038cfca2cf4856df3797bdd9"}
Sep 18 19:17:32.324455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:32.324342    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd46f7b7-jl6rk" podStartSLOduration=1.3243267429999999 podStartE2EDuration="1.324326743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:32.32312674 +0000 UTC m=+213.110407145" watchObservedRunningTime="2026-09-18 19:17:32.324326743 +0000 UTC m=+213.111607146"
Sep 18 19:17:41.778946 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:41.778904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:41.779406 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:41.778995    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:41.783365 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:41.783329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:42.345538 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:42.345510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:17:42.404725 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:42.404685    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:17:42.550547 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:42.550509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:42.650911 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:42.650805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:42.724294 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:17:42.724254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:07.426419 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.426371    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c9fc4cb77-nst4t" podUID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" containerName="console" containerID="cri-o://f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0" gracePeriod=15
Sep 18 19:18:07.675705 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.675681    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:18:07.833527 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdkxr\" (UniqueName: \"kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833752 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.833999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.833771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config\") pod \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\" (UID: \"9313224b-4aa0-4a4f-b580-68c1b3dd44dd\") "
Sep 18 19:18:07.834072 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.834031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:07.834191 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.834162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:07.834260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.834218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:07.834315 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.834281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config" (OuterVolumeSpecName: "console-config") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:07.835869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.835841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:07.835869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.835858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:07.836020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.835925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr" (OuterVolumeSpecName: "kube-api-access-tdkxr") pod "9313224b-4aa0-4a4f-b580-68c1b3dd44dd" (UID: "9313224b-4aa0-4a4f-b580-68c1b3dd44dd"). InnerVolumeSpecName "kube-api-access-tdkxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:18:07.934493 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934463    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934493 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934487    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-oauth-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934493 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934499    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-trusted-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934508    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-console-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934517    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdkxr\" (UniqueName: \"kubernetes.io/projected/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-kube-api-access-tdkxr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934525    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-oauth-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:07.934750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:07.934532    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9313224b-4aa0-4a4f-b580-68c1b3dd44dd-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:18:08.445565 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.445531    2580 generic.go:356] "Generic (PLEG): container finished" podID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" containerID="f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0" exitCode=0
Sep 18 19:18:08.445999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.445594    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fc4cb77-nst4t"
Sep 18 19:18:08.445999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.445621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fc4cb77-nst4t" event={"ID":"9313224b-4aa0-4a4f-b580-68c1b3dd44dd","Type":"ContainerDied","Data":"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"}
Sep 18 19:18:08.445999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.445678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fc4cb77-nst4t" event={"ID":"9313224b-4aa0-4a4f-b580-68c1b3dd44dd","Type":"ContainerDied","Data":"451f0c099b411eed5866708cde549dc308e8afa77ca531d5052e779ed4000e0f"}
Sep 18 19:18:08.445999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.445705    2580 scope.go:117] "RemoveContainer" containerID="f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"
Sep 18 19:18:08.454814 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.454795    2580 scope.go:117] "RemoveContainer" containerID="f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"
Sep 18 19:18:08.455112 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:18:08.455095    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0\": container with ID starting with f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0 not found: ID does not exist" containerID="f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"
Sep 18 19:18:08.455163 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.455121    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0"} err="failed to get container status \"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0\": rpc error: code = NotFound desc = could not find container \"f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0\": container with ID starting with f6d5b528175db83f23ab4f5b6fbb05e0b064ee9cd479424b40dd4b7e84378db0 not found: ID does not exist"
Sep 18 19:18:08.478233 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.478202    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:18:08.489573 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:08.489541    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c9fc4cb77-nst4t"]
Sep 18 19:18:09.293022 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:09.292989    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" path="/var/lib/kubelet/pods/9313224b-4aa0-4a4f-b580-68c1b3dd44dd/volumes"
Sep 18 19:18:16.768104 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:16.768031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:16.984979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:16.984953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:37.892616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:37.892565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:38.136235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:38.136201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:49.818843 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:49.818799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:49.954290 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:49.954247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:52.904739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:52.904693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:52.999602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:52.999550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:53.099615 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:53.099559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:59.107768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.107729    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:18:59.108162 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.108051    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" containerName="console"
Sep 18 19:18:59.108162 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.108062    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" containerName="console"
Sep 18 19:18:59.108162 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.108116    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9313224b-4aa0-4a4f-b580-68c1b3dd44dd" containerName="console"
Sep 18 19:18:59.110986 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.110969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.140873 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.140842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:18:59.254296 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254469 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254469 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254469 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254469 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254658 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghvz\" (UniqueName: \"kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.254658 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.254504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghvz\" (UniqueName: \"kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.355909 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.355689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.356194 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.356160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.356365 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.356346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.356434 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.356377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.356434 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.356380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.358161 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.358108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.358235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.358194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.383737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.383712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:59.390921 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.390899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghvz\" (UniqueName: \"kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz\") pod \"console-6d9fc45b64-gplnb\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") " pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.420058 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.420026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:18:59.489053 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.489024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:59.702044 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.701957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:18:59.705747 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:18:59.705718    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a729e6f_5ecd_4e07_af4a_4f7588f80b67.slice/crio-2bcd35f506386a1d22d5e1f7b3b516f34dd8ab423b8670fcc3cac88cf0e64f82 WatchSource:0}: Error finding container 2bcd35f506386a1d22d5e1f7b3b516f34dd8ab423b8670fcc3cac88cf0e64f82: Status 404 returned error can't find the container with id 2bcd35f506386a1d22d5e1f7b3b516f34dd8ab423b8670fcc3cac88cf0e64f82
Sep 18 19:18:59.826423 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.826386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:59.910608 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:18:59.910560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:00.649606 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:00.649548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9fc45b64-gplnb" event={"ID":"3a729e6f-5ecd-4e07-af4a-4f7588f80b67","Type":"ContainerStarted","Data":"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"}
Sep 18 19:19:00.650047 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:00.649619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9fc45b64-gplnb" event={"ID":"3a729e6f-5ecd-4e07-af4a-4f7588f80b67","Type":"ContainerStarted","Data":"2bcd35f506386a1d22d5e1f7b3b516f34dd8ab423b8670fcc3cac88cf0e64f82"}
Sep 18 19:19:00.673513 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:00.673452    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d9fc45b64-gplnb" podStartSLOduration=1.673435908 podStartE2EDuration="1.673435908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:19:00.671574428 +0000 UTC m=+301.458854832" watchObservedRunningTime="2026-09-18 19:19:00.673435908 +0000 UTC m=+301.460716311"
Sep 18 19:19:01.173773 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:01.173747    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:19:01.178833 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:01.178807    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:19:01.181183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:01.181159    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:19:01.755248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:01.755210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:08.563152 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:08.563120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:09.421032 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.420990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:19:09.421214 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.421069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:19:09.425668 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.425642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:19:09.687071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.686994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:19:09.783937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.783906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:19:09.970917 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:09.970826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:10.072273 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:10.072240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:14.404255 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:14.404218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:31.557813 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:31.557762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:31.654659 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:31.654611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:31.719722 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:31.719690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:34.802700 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:34.802662    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd46f7b7-jl6rk" podUID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" containerName="console" containerID="cri-o://5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff" gracePeriod=15
Sep 18 19:19:35.044446 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.044423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:19:35.048885 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.048860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049022 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.048968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmg55\" (UniqueName: \"kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049022 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049126    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert\") pod \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\" (UID: \"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50\") "
Sep 18 19:19:35.049443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca" (OuterVolumeSpecName: "service-ca") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:35.049597 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config" (OuterVolumeSpecName: "console-config") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:35.049597 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:35.049853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.049744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:35.051338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.051313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:35.051542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.051518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:35.051661 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.051539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55" (OuterVolumeSpecName: "kube-api-access-wmg55") pod "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" (UID: "f0bc8edb-16db-4c3b-b7f6-75d0556e2d50"). InnerVolumeSpecName "kube-api-access-wmg55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:35.149977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.149937    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.149977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.149969    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmg55\" (UniqueName: \"kubernetes.io/projected/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-kube-api-access-wmg55\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.149977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.149979    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.149977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.149988    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-trusted-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.150260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.149996    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.150260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.150006    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-console-oauth-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.150260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.150014    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50-oauth-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:19:35.788260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.788225    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" containerID="5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff" exitCode=0
Sep 18 19:19:35.788457 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.788270    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd46f7b7-jl6rk"
Sep 18 19:19:35.788457 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.788311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd46f7b7-jl6rk" event={"ID":"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50","Type":"ContainerDied","Data":"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"}
Sep 18 19:19:35.788457 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.788363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd46f7b7-jl6rk" event={"ID":"f0bc8edb-16db-4c3b-b7f6-75d0556e2d50","Type":"ContainerDied","Data":"851c2f3248c150b46bd5430d9982b3c1b4fee6d1038cfca2cf4856df3797bdd9"}
Sep 18 19:19:35.788457 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.788384    2580 scope.go:117] "RemoveContainer" containerID="5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"
Sep 18 19:19:35.796374 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.796357    2580 scope.go:117] "RemoveContainer" containerID="5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"
Sep 18 19:19:35.796672 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:19:35.796645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff\": container with ID starting with 5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff not found: ID does not exist" containerID="5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"
Sep 18 19:19:35.796736 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.796674    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff"} err="failed to get container status \"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff\": rpc error: code = NotFound desc = could not find container \"5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff\": container with ID starting with 5139a0c3bcc92d62fd1ec56b3d133b49f7a4c45ac2dd0e968fd0ab1c0eebe7ff not found: ID does not exist"
Sep 18 19:19:35.818572 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.818538    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:19:35.826675 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:35.826645    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd46f7b7-jl6rk"]
Sep 18 19:19:37.294150 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:19:37.294114    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" path="/var/lib/kubelet/pods/f0bc8edb-16db-4c3b-b7f6-75d0556e2d50/volumes"
Sep 18 19:20:53.451682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:20:53.451642    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:54.548404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:20:54.548375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:59.466604 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:20:59.466546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:08.660840 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:08.660807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:08.803015 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:08.802988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:09.572015 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.571983    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f86c8bf58-8b62j"]
Sep 18 19:22:09.572293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.572281    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" containerName="console"
Sep 18 19:22:09.572338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.572295    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" containerName="console"
Sep 18 19:22:09.572376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.572344    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bc8edb-16db-4c3b-b7f6-75d0556e2d50" containerName="console"
Sep 18 19:22:09.575148 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.575134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.678611 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.678570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f86c8bf58-8b62j"]
Sep 18 19:22:09.682210 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklvl\" (UniqueName: \"kubernetes.io/projected/ebe9c100-5cd9-4beb-9655-2c30dcc09401-kube-api-access-jklvl\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-service-ca\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-oauth-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-trusted-ca-bundle\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.682460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.682392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-oauth-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.766157 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.766128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:09.783262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-oauth-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783367 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783367 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-trusted-ca-bundle\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783367 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-oauth-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklvl\" (UniqueName: \"kubernetes.io/projected/ebe9c100-5cd9-4beb-9655-2c30dcc09401-kube-api-access-jklvl\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.783542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.783455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-service-ca\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.784071 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.784043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.784201 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.784043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-oauth-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.784265 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.784251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-service-ca\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.784521 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.784499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe9c100-5cd9-4beb-9655-2c30dcc09401-trusted-ca-bundle\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.785719 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.785693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-serving-cert\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.785803 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.785752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebe9c100-5cd9-4beb-9655-2c30dcc09401-console-oauth-config\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.802982 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.802956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklvl\" (UniqueName: \"kubernetes.io/projected/ebe9c100-5cd9-4beb-9655-2c30dcc09401-kube-api-access-jklvl\") pod \"console-f86c8bf58-8b62j\" (UID: \"ebe9c100-5cd9-4beb-9655-2c30dcc09401\") " pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:09.884674 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:09.884597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:10.180261 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:10.180148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f86c8bf58-8b62j"]
Sep 18 19:22:10.182829 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:22:10.182802    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe9c100_5cd9_4beb_9655_2c30dcc09401.slice/crio-9294cd37bd8c6ffc9defe43ceb7e3548d876e84ea259446e657a05ede68ccfa1 WatchSource:0}: Error finding container 9294cd37bd8c6ffc9defe43ceb7e3548d876e84ea259446e657a05ede68ccfa1: Status 404 returned error can't find the container with id 9294cd37bd8c6ffc9defe43ceb7e3548d876e84ea259446e657a05ede68ccfa1
Sep 18 19:22:10.383683 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:10.383646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f86c8bf58-8b62j" event={"ID":"ebe9c100-5cd9-4beb-9655-2c30dcc09401","Type":"ContainerStarted","Data":"696270fbf7ff1ce74cd80789816e2bd093a8717e5d879fead75dd611947beb36"}
Sep 18 19:22:10.383847 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:10.383704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f86c8bf58-8b62j" event={"ID":"ebe9c100-5cd9-4beb-9655-2c30dcc09401","Type":"ContainerStarted","Data":"9294cd37bd8c6ffc9defe43ceb7e3548d876e84ea259446e657a05ede68ccfa1"}
Sep 18 19:22:10.419825 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:10.419778    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f86c8bf58-8b62j" podStartSLOduration=1.419762685 podStartE2EDuration="1.419762685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:10.413858029 +0000 UTC m=+491.201138434" watchObservedRunningTime="2026-09-18 19:22:10.419762685 +0000 UTC m=+491.207043088"
Sep 18 19:22:11.263154 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.263122    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"]
Sep 18 19:22:11.266413 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.266392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.273180 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.273067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-5sbqb\""
Sep 18 19:22:11.273291 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.273222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:22:11.273396 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.273365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:22:11.273641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.273623    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:22:11.283851 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.283832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"]
Sep 18 19:22:11.398885 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.398856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwff\" (UniqueName: \"kubernetes.io/projected/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-kube-api-access-jdwff\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.399046 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.398956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.499849 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.499813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwff\" (UniqueName: \"kubernetes.io/projected/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-kube-api-access-jdwff\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.500067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.500049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.502244 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.502223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.510281 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.510247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwff\" (UniqueName: \"kubernetes.io/projected/ee8ed321-9fd5-4dec-8466-0c3e56b04e0d-kube-api-access-jdwff\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-k9kdb\" (UID: \"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.577280 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.577255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:11.878933 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.878909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"]
Sep 18 19:22:11.880679 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:22:11.880648    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8ed321_9fd5_4dec_8466_0c3e56b04e0d.slice/crio-bd4b2a35804678412c65cb37beecd810f8164faeb2c5f46f006592792a5e02fd WatchSource:0}: Error finding container bd4b2a35804678412c65cb37beecd810f8164faeb2c5f46f006592792a5e02fd: Status 404 returned error can't find the container with id bd4b2a35804678412c65cb37beecd810f8164faeb2c5f46f006592792a5e02fd
Sep 18 19:22:11.882701 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:11.882681    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:22:12.392312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:12.392278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb" event={"ID":"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d","Type":"ContainerStarted","Data":"bd4b2a35804678412c65cb37beecd810f8164faeb2c5f46f006592792a5e02fd"}
Sep 18 19:22:18.994318 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:18.994280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:19.422612 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.422558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb" event={"ID":"ee8ed321-9fd5-4dec-8466-0c3e56b04e0d","Type":"ContainerStarted","Data":"4151ee648213ea1887794f9145e6777b7773c2d3073dadc534c774f120018433"}
Sep 18 19:22:19.422775 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.422724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:19.482088 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.482042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb" podStartSLOduration=1.9144504530000002 podStartE2EDuration="8.482027884s" totalImagesPullingTime="6.567577431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:19.480859242 +0000 UTC m=+500.268139645" watchObservedRunningTime="2026-09-18 19:22:19.482027884 +0000 UTC m=+500.269308288"
Sep 18 19:22:19.659870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.659841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:19.679635 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.679556    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vqm7d"]
Sep 18 19:22:19.682804 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.682790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.685880 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.685862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 19:22:19.686337 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.686321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lbv7p\""
Sep 18 19:22:19.686394 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.686357    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:22:19.708943 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.708918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vqm7d"]
Sep 18 19:22:19.771240 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.771209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d5nb\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-kube-api-access-4d5nb\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.771390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.771255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/879e6f1e-05f7-4616-89df-c4cff63b2950-cabundle0\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.771390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.771334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.840314 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.840283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:19.872262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.872237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.872328 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.872278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d5nb\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-kube-api-access-4d5nb\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.872328 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.872308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/879e6f1e-05f7-4616-89df-c4cff63b2950-cabundle0\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.872420 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:19.872374    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:22:19.872420 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:19.872391    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:22:19.872420 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:19.872399    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-vqm7d: references non-existent secret key: ca.crt
Sep 18 19:22:19.872519 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:19.872448    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates podName:879e6f1e-05f7-4616-89df-c4cff63b2950 nodeName:}" failed. No retries permitted until 2026-09-18 19:22:20.372433599 +0000 UTC m=+501.159713981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates") pod "keda-operator-579c8958f7-vqm7d" (UID: "879e6f1e-05f7-4616-89df-c4cff63b2950") : references non-existent secret key: ca.crt
Sep 18 19:22:19.872898 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.872882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/879e6f1e-05f7-4616-89df-c4cff63b2950-cabundle0\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.885087 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.885067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:19.885180 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.885108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:19.887925 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.887898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d5nb\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-kube-api-access-4d5nb\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:19.889975 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:19.889949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:20.377929 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:20.377870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:20.378368 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:20.377998    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:22:20.378368 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:20.378018    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:22:20.378368 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:20.378030    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-vqm7d: references non-existent secret key: ca.crt
Sep 18 19:22:20.378368 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:20.378105    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates podName:879e6f1e-05f7-4616-89df-c4cff63b2950 nodeName:}" failed. No retries permitted until 2026-09-18 19:22:21.378085181 +0000 UTC m=+502.165365563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates") pod "keda-operator-579c8958f7-vqm7d" (UID: "879e6f1e-05f7-4616-89df-c4cff63b2950") : references non-existent secret key: ca.crt
Sep 18 19:22:20.431363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:20.431337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f86c8bf58-8b62j"
Sep 18 19:22:20.521058 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:20.521018    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:22:20.741442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:20.741360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:20.933061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:20.933026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.060420 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.060377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.178048 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.178017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.370842 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.370757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.386850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.386820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:21.389403 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.389374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/879e6f1e-05f7-4616-89df-c4cff63b2950-certificates\") pod \"keda-operator-579c8958f7-vqm7d\" (UID: \"879e6f1e-05f7-4616-89df-c4cff63b2950\") " pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:21.479692 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.479651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.493299 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.493271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:21.639101 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.639028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:21.775853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.775825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vqm7d"]
Sep 18 19:22:21.779252 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:22:21.779221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879e6f1e_05f7_4616_89df_c4cff63b2950.slice/crio-d794051e0ebee07225ae387ff8ab21d776c80eb6b096295c657b3e8a980e53a7 WatchSource:0}: Error finding container d794051e0ebee07225ae387ff8ab21d776c80eb6b096295c657b3e8a980e53a7: Status 404 returned error can't find the container with id d794051e0ebee07225ae387ff8ab21d776c80eb6b096295c657b3e8a980e53a7
Sep 18 19:22:21.828993 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:21.828967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:22.017603 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:22.017496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:22.436970 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:22.436936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-vqm7d" event={"ID":"879e6f1e-05f7-4616-89df-c4cff63b2950","Type":"ContainerStarted","Data":"d794051e0ebee07225ae387ff8ab21d776c80eb6b096295c657b3e8a980e53a7"}
Sep 18 19:22:29.467602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:29.467552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-vqm7d" event={"ID":"879e6f1e-05f7-4616-89df-c4cff63b2950","Type":"ContainerStarted","Data":"bc99342ac7e49ede1e52937d6f22a417dd79aabd8f69ac170d897cf6fe603423"}
Sep 18 19:22:29.467958 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:29.467711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:22:29.510557 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:29.510510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-vqm7d" podStartSLOduration=3.503016561 podStartE2EDuration="10.510497133s" totalImagesPullingTime="7.007480572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:29.507780854 +0000 UTC m=+510.295061258" watchObservedRunningTime="2026-09-18 19:22:29.510497133 +0000 UTC m=+510.297777536"
Sep 18 19:22:40.429811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:40.429781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-k9kdb"
Sep 18 19:22:45.542104 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.542028    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d9fc45b64-gplnb" podUID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" containerName="console" containerID="cri-o://d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56" gracePeriod=15
Sep 18 19:22:45.800489 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.800469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:22:45.892395 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jghvz\" (UniqueName: \"kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config\") pod \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\" (UID: \"3a729e6f-5ecd-4e07-af4a-4f7588f80b67\") "
Sep 18 19:22:45.892953 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:45.893011 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.892946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:45.893055 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.893018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config" (OuterVolumeSpecName: "console-config") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:45.893091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.893066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:22:45.894681 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.894656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:22:45.894949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.894930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:22:45.894949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.894937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz" (OuterVolumeSpecName: "kube-api-access-jghvz") pod "3a729e6f-5ecd-4e07-af4a-4f7588f80b67" (UID: "3a729e6f-5ecd-4e07-af4a-4f7588f80b67"). InnerVolumeSpecName "kube-api-access-jghvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:22:45.993354 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993319    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-oauth-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993354 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993347    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-trusted-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993354 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993359    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993576 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993371    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993576 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghvz\" (UniqueName: \"kubernetes.io/projected/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-kube-api-access-jghvz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993576 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993393    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-oauth-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:45.993576 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:45.993404    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a729e6f-5ecd-4e07-af4a-4f7588f80b67-console-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:22:46.542828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.542800    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" containerID="d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56" exitCode=0
Sep 18 19:22:46.542828 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.542834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9fc45b64-gplnb" event={"ID":"3a729e6f-5ecd-4e07-af4a-4f7588f80b67","Type":"ContainerDied","Data":"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"}
Sep 18 19:22:46.543241 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.542849    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9fc45b64-gplnb"
Sep 18 19:22:46.543241 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.542870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9fc45b64-gplnb" event={"ID":"3a729e6f-5ecd-4e07-af4a-4f7588f80b67","Type":"ContainerDied","Data":"2bcd35f506386a1d22d5e1f7b3b516f34dd8ab423b8670fcc3cac88cf0e64f82"}
Sep 18 19:22:46.543241 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.542891    2580 scope.go:117] "RemoveContainer" containerID="d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"
Sep 18 19:22:46.551806 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.551783    2580 scope.go:117] "RemoveContainer" containerID="d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"
Sep 18 19:22:46.552083 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:22:46.552063    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56\": container with ID starting with d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56 not found: ID does not exist" containerID="d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"
Sep 18 19:22:46.552150 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.552090    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56"} err="failed to get container status \"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56\": rpc error: code = NotFound desc = could not find container \"d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56\": container with ID starting with d35c2456ef74fcc58ca6fc834074fbf727501c51fb6361b351c70e54091c9f56 not found: ID does not exist"
Sep 18 19:22:46.568174 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.568150    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:22:46.576471 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:46.576450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9fc45b64-gplnb"]
Sep 18 19:22:47.294339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:47.294310    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" path="/var/lib/kubelet/pods/3a729e6f-5ecd-4e07-af4a-4f7588f80b67/volumes"
Sep 18 19:22:50.474436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:22:50.474401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-vqm7d"
Sep 18 19:23:12.252942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:12.252909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:12.354577 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:12.354542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:23.661792 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:23.661736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:24.569646 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:24.569610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:26.071373 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.071336    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"]
Sep 18 19:23:26.071753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.071680    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" containerName="console"
Sep 18 19:23:26.071753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.071690    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" containerName="console"
Sep 18 19:23:26.071753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.071754    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a729e6f-5ecd-4e07-af4a-4f7588f80b67" containerName="console"
Sep 18 19:23:26.073774 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.073758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.076948 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.076931    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 19:23:26.077279 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.077266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:23:26.077506 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.077491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xkt2d\""
Sep 18 19:23:26.098330 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.098306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"]
Sep 18 19:23:26.214682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.214651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d208940-dad1-4f46-a6bb-037baaf6c6e3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.214878 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.214688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28r2\" (UniqueName: \"kubernetes.io/projected/7d208940-dad1-4f46-a6bb-037baaf6c6e3-kube-api-access-f28r2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.316062 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.316034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d208940-dad1-4f46-a6bb-037baaf6c6e3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.316062 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.316067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28r2\" (UniqueName: \"kubernetes.io/projected/7d208940-dad1-4f46-a6bb-037baaf6c6e3-kube-api-access-f28r2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.316478 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.316455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d208940-dad1-4f46-a6bb-037baaf6c6e3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.325754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.325687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28r2\" (UniqueName: \"kubernetes.io/projected/7d208940-dad1-4f46-a6bb-037baaf6c6e3-kube-api-access-f28r2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-p9gvh\" (UID: \"7d208940-dad1-4f46-a6bb-037baaf6c6e3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.383214 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.383189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"
Sep 18 19:23:26.673054 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.672976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh"]
Sep 18 19:23:26.677195 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:23:26.677169    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d208940_dad1_4f46_a6bb_037baaf6c6e3.slice/crio-45b35a7416a60845c676fbe65b2e5ecd7e92dda943ae76b883845b3142e45687 WatchSource:0}: Error finding container 45b35a7416a60845c676fbe65b2e5ecd7e92dda943ae76b883845b3142e45687: Status 404 returned error can't find the container with id 45b35a7416a60845c676fbe65b2e5ecd7e92dda943ae76b883845b3142e45687
Sep 18 19:23:26.714946 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:26.714915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh" event={"ID":"7d208940-dad1-4f46-a6bb-037baaf6c6e3","Type":"ContainerStarted","Data":"45b35a7416a60845c676fbe65b2e5ecd7e92dda943ae76b883845b3142e45687"}
Sep 18 19:23:27.879304 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:27.879270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:27.985424 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:27.985369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:29.732446 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:29.732408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh" event={"ID":"7d208940-dad1-4f46-a6bb-037baaf6c6e3","Type":"ContainerStarted","Data":"43bbc1132a9108ba1d8ec4395b71cd8b06b025434d1f146f28c33891af30fcb4"}
Sep 18 19:23:29.769661 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:29.769594    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-p9gvh" podStartSLOduration=1.5149524300000001 podStartE2EDuration="3.769563411s" totalImagesPullingTime="2.254610981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:29.766481892 +0000 UTC m=+570.553762295" watchObservedRunningTime="2026-09-18 19:23:29.769563411 +0000 UTC m=+570.556843815"
Sep 18 19:23:30.090442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:30.090411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:30.308139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:30.308094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:32.510801 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:32.510772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:32.688734 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:32.688701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:33.876017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:33.875978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:34.017349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:34.017310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:35.370116 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:35.370083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:35.553294 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:35.553263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:37.413009 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.412974    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nn74w"]
Sep 18 19:23:37.415412 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.415395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.419263 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.419246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-n6jxt\""
Sep 18 19:23:37.420958 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.420939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 19:23:37.420958 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.420955    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 19:23:37.429336 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.429312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nn74w"]
Sep 18 19:23:37.509242 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.509205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.509410 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.509292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cpx\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-kube-api-access-q5cpx\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.610351 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.610311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.610512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.610364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cpx\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-kube-api-access-q5cpx\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.623304 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.623271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.623425 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.623349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cpx\" (UniqueName: \"kubernetes.io/projected/f243c812-dd31-4d88-9a8c-73bced39c381-kube-api-access-q5cpx\") pod \"cert-manager-webhook-6486ffd997-nn74w\" (UID: \"f243c812-dd31-4d88-9a8c-73bced39c381\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:37.735524 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:37.735430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:38.006369 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:38.006236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nn74w"]
Sep 18 19:23:38.009139 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:23:38.009116    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf243c812_dd31_4d88_9a8c_73bced39c381.slice/crio-dd570d067cf6a64ff9e64d49b0b581de7a65dc8a1657dbbd77b0b435fc0178a0 WatchSource:0}: Error finding container dd570d067cf6a64ff9e64d49b0b581de7a65dc8a1657dbbd77b0b435fc0178a0: Status 404 returned error can't find the container with id dd570d067cf6a64ff9e64d49b0b581de7a65dc8a1657dbbd77b0b435fc0178a0
Sep 18 19:23:38.771769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:38.771738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w" event={"ID":"f243c812-dd31-4d88-9a8c-73bced39c381","Type":"ContainerStarted","Data":"dd570d067cf6a64ff9e64d49b0b581de7a65dc8a1657dbbd77b0b435fc0178a0"}
Sep 18 19:23:40.035702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.035665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:40.638674 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.638575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:40.782090 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.782059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w" event={"ID":"f243c812-dd31-4d88-9a8c-73bced39c381","Type":"ContainerStarted","Data":"3a3bb3845902871e2b16b2de637860df4ae72f167b41b49ff246058687efed5f"}
Sep 18 19:23:40.782263 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.782121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:23:40.795109 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.795084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:40.832447 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:40.832395    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w" podStartSLOduration=1.494285608 podStartE2EDuration="3.832380879s" totalImagesPullingTime="2.338095271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:40.831762962 +0000 UTC m=+581.619043365" watchObservedRunningTime="2026-09-18 19:23:40.832380879 +0000 UTC m=+581.619661280"
Sep 18 19:23:43.484185 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:43.484149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:43.577020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:43.576987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:44.731194 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.731159    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v7tgl"]
Sep 18 19:23:44.733353 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.733338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.736635 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.736619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vwcq6\""
Sep 18 19:23:44.745160 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.745139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v7tgl"]
Sep 18 19:23:44.769022 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.768989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl947\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-kube-api-access-kl947\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.769191 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.769041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-bound-sa-token\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.870274 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.870243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl947\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-kube-api-access-kl947\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.870463 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.870292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-bound-sa-token\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.879706 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.879680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-bound-sa-token\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:44.879971 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:44.879950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl947\" (UniqueName: \"kubernetes.io/projected/b72be18e-1189-4d67-9295-ea797ad9d798-kube-api-access-kl947\") pod \"cert-manager-55b4d954d5-v7tgl\" (UID: \"b72be18e-1189-4d67-9295-ea797ad9d798\") " pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:45.042961 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:45.042881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v7tgl"
Sep 18 19:23:45.314647 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:45.314619    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v7tgl"]
Sep 18 19:23:45.318682 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:23:45.318646    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72be18e_1189_4d67_9295_ea797ad9d798.slice/crio-89ec98d96017d314f5994a7d26c51f165f13c519c4f49f96dbf4a975cd8d1599 WatchSource:0}: Error finding container 89ec98d96017d314f5994a7d26c51f165f13c519c4f49f96dbf4a975cd8d1599: Status 404 returned error can't find the container with id 89ec98d96017d314f5994a7d26c51f165f13c519c4f49f96dbf4a975cd8d1599
Sep 18 19:23:45.804141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:45.804109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v7tgl" event={"ID":"b72be18e-1189-4d67-9295-ea797ad9d798","Type":"ContainerStarted","Data":"cea804a4542b50e27a589c58be8629b8436d7a909309b87ce29f1b5c6edf276c"}
Sep 18 19:23:45.804501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:45.804150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v7tgl" event={"ID":"b72be18e-1189-4d67-9295-ea797ad9d798","Type":"ContainerStarted","Data":"89ec98d96017d314f5994a7d26c51f165f13c519c4f49f96dbf4a975cd8d1599"}
Sep 18 19:23:45.823301 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:45.823250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-v7tgl" podStartSLOduration=1.823234843 podStartE2EDuration="1.823234843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:45.822174747 +0000 UTC m=+586.609455150" watchObservedRunningTime="2026-09-18 19:23:45.823234843 +0000 UTC m=+586.610515245"
Sep 18 19:23:46.788508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:23:46.788477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nn74w"
Sep 18 19:24:01.203699 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:01.203668    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:24:01.206339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:01.206317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:24:02.975815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:02.975787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:07.382838 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:07.382784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:07.494310 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:07.494279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:08.409990 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.409950    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:24:08.412295 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.412280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.416651 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416611    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.416651 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416615    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cacerts\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret"
Sep 18 19:24:08.416845 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416662    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-tls\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret"
Sep 18 19:24:08.416903 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416839    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istio-kubeconfig\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret"
Sep 18 19:24:08.416957 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416915    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap"
Sep 18 19:24:08.416957 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:08.416926    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-openshift-gateway-dockercfg-2477j\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2477j\"" type="*v1.Secret"
Sep 18 19:24:08.429903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.429878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:08.478063 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478205 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478205 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478205 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2g42\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478205 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478397 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.478397 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.478287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.497106 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.497082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:24:08.578921 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.578887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.578921 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.578921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.579139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.578969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.579139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.578998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.579139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.579013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.579139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.579040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2g42\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.579139 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.579067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.581289 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.581268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.594903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.594875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2g42\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.609649 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.609622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:08.859621 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:08.859564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:09.377155 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.377126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 19:24:09.381273 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.381251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:09.570380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.570349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:24:09.580098 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:09.580076    2580 secret.go:189] Couldn't get secret openshift-ingress/istiod-tls: failed to sync secret cache: timed out waiting for the condition
Sep 18 19:24:09.580211 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:09.580150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert podName:99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc nodeName:}" failed. No retries permitted until 2026-09-18 19:24:10.080135536 +0000 UTC m=+610.867415921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-csr-dns-cert" (UniqueName: "kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert") pod "istiod-openshift-gateway-7b7995d5cd-4x4m6" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc") : failed to sync secret cache: timed out waiting for the condition
Sep 18 19:24:09.580211 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:09.580076    2580 secret.go:189] Couldn't get secret openshift-ingress/cacerts: failed to sync secret cache: timed out waiting for the condition
Sep 18 19:24:09.580332 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:24:09.580240    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts podName:99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc nodeName:}" failed. No retries permitted until 2026-09-18 19:24:10.080223333 +0000 UTC m=+610.867503716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cacerts" (UniqueName: "kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts") pod "istiod-openshift-gateway-7b7995d5cd-4x4m6" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc") : failed to sync secret cache: timed out waiting for the condition
Sep 18 19:24:09.580574 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.580553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:09.661184 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.661118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 19:24:09.713108 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.713081    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 19:24:09.785100 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:09.785073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2477j\""
Sep 18 19:24:10.089911 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.089871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:10.090077 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.089928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:10.092189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.092163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:10.092288 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.092215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-4x4m6\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:10.221109 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.221081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:10.554232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.554210    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:24:10.557800 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:24:10.557770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f2e2a1_6401_4972_af6c_ea2d1fb3a6cc.slice/crio-d367a70345622a1f0a11fc21ef5e34a52827e77bd642c0b508a34255879dd198 WatchSource:0}: Error finding container d367a70345622a1f0a11fc21ef5e34a52827e77bd642c0b508a34255879dd198: Status 404 returned error can't find the container with id d367a70345622a1f0a11fc21ef5e34a52827e77bd642c0b508a34255879dd198
Sep 18 19:24:10.920616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:10.920517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" event={"ID":"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc","Type":"ContainerStarted","Data":"d367a70345622a1f0a11fc21ef5e34a52827e77bd642c0b508a34255879dd198"}
Sep 18 19:24:11.343727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:11.343690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:13.241384 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:13.241333    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:24:13.241672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:13.241419    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:24:13.938142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:13.938105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" event={"ID":"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc","Type":"ContainerStarted","Data":"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"}
Sep 18 19:24:13.938348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:13.938202    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:13.965100 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:13.965053    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" podStartSLOduration=3.284150995 podStartE2EDuration="5.965038415s" totalImagesPullingTime="2.68088742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:13.963033391 +0000 UTC m=+614.750313796" watchObservedRunningTime="2026-09-18 19:24:13.965038415 +0000 UTC m=+614.752318817"
Sep 18 19:24:14.179476 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:14.179427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:14.279643 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:14.279524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:14.747336 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:14.747307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:14.944136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:14.944105    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-4x4m6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:24:14.944281 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:14.944151    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:24:17.943727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:17.943696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:24:18.539464 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:18.539429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:18.610674 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:18.610636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:21.980525 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:21.980492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:24.246388 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:24.246355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:24.348442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:24.348402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:25.785407 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:25.785376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:38.005543 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:38.005507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:38.791385 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:38.791341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:41.122528 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.122495    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"]
Sep 18 19:24:41.126422 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.126405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:41.130596 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.130562    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bsrdq\""
Sep 18 19:24:41.131017 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.131002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:24:41.132126 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.132109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:24:41.141509 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.141484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9nl\" (UniqueName: \"kubernetes.io/projected/cee93929-03a4-4077-96ad-0809011aa6f3-kube-api-access-4h9nl\") pod \"authorino-operator-6fc8f864f4-wn4r4\" (UID: \"cee93929-03a4-4077-96ad-0809011aa6f3\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:41.141933 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.141910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"]
Sep 18 19:24:41.242127 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.242091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9nl\" (UniqueName: \"kubernetes.io/projected/cee93929-03a4-4077-96ad-0809011aa6f3-kube-api-access-4h9nl\") pod \"authorino-operator-6fc8f864f4-wn4r4\" (UID: \"cee93929-03a4-4077-96ad-0809011aa6f3\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:41.256150 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.256122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9nl\" (UniqueName: \"kubernetes.io/projected/cee93929-03a4-4077-96ad-0809011aa6f3-kube-api-access-4h9nl\") pod \"authorino-operator-6fc8f864f4-wn4r4\" (UID: \"cee93929-03a4-4077-96ad-0809011aa6f3\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:41.291942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.291913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:41.424805 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.424718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:41.436731 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.436707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:41.717529 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.717503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"]
Sep 18 19:24:41.719164 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:24:41.719138    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee93929_03a4_4077_96ad_0809011aa6f3.slice/crio-30b1dbc5077536b65a995e5da62a6593a65ce51bbc17e2ffe2d7e75b086e991e WatchSource:0}: Error finding container 30b1dbc5077536b65a995e5da62a6593a65ce51bbc17e2ffe2d7e75b086e991e: Status 404 returned error can't find the container with id 30b1dbc5077536b65a995e5da62a6593a65ce51bbc17e2ffe2d7e75b086e991e
Sep 18 19:24:41.989174 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:41.989090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:42.063289 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:42.063258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4" event={"ID":"cee93929-03a4-4077-96ad-0809011aa6f3","Type":"ContainerStarted","Data":"30b1dbc5077536b65a995e5da62a6593a65ce51bbc17e2ffe2d7e75b086e991e"}
Sep 18 19:24:43.344979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.344951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"]
Sep 18 19:24:43.347393 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.347365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:43.350433 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.350413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5s9v4\""
Sep 18 19:24:43.359763 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.359744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg52\" (UniqueName: \"kubernetes.io/projected/395caf8f-4cd7-4c55-bed4-3219dcf1ee3c-kube-api-access-rzg52\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8ndkz\" (UID: \"395caf8f-4cd7-4c55-bed4-3219dcf1ee3c\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:43.370217 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.370194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"]
Sep 18 19:24:43.416152 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.416125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:43.461140 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.461063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg52\" (UniqueName: \"kubernetes.io/projected/395caf8f-4cd7-4c55-bed4-3219dcf1ee3c-kube-api-access-rzg52\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8ndkz\" (UID: \"395caf8f-4cd7-4c55-bed4-3219dcf1ee3c\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:43.479232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.479207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg52\" (UniqueName: \"kubernetes.io/projected/395caf8f-4cd7-4c55-bed4-3219dcf1ee3c-kube-api-access-rzg52\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8ndkz\" (UID: \"395caf8f-4cd7-4c55-bed4-3219dcf1ee3c\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:43.656827 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.656796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:43.960391 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:43.960363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"]
Sep 18 19:24:43.963375 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:24:43.963343    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395caf8f_4cd7_4c55_bed4_3219dcf1ee3c.slice/crio-e46bc0039a0cce861fb57d141553cfa8757033aea31678076ebc835b5fc2aa61 WatchSource:0}: Error finding container e46bc0039a0cce861fb57d141553cfa8757033aea31678076ebc835b5fc2aa61: Status 404 returned error can't find the container with id e46bc0039a0cce861fb57d141553cfa8757033aea31678076ebc835b5fc2aa61
Sep 18 19:24:44.076508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:44.076472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz" event={"ID":"395caf8f-4cd7-4c55-bed4-3219dcf1ee3c","Type":"ContainerStarted","Data":"e46bc0039a0cce861fb57d141553cfa8757033aea31678076ebc835b5fc2aa61"}
Sep 18 19:24:44.078190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:44.078166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4" event={"ID":"cee93929-03a4-4077-96ad-0809011aa6f3","Type":"ContainerStarted","Data":"83bd03a954b2cee8280468c543c7082ddfedc2c6cc2d9a9f4834b52077bdbe7e"}
Sep 18 19:24:44.078306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:44.078218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:44.111375 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:44.111332    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4" podStartSLOduration=1.6236612780000002 podStartE2EDuration="3.111317303s" totalImagesPullingTime="1.487656025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:44.110906507 +0000 UTC m=+644.898186913" watchObservedRunningTime="2026-09-18 19:24:44.111317303 +0000 UTC m=+644.898597705"
Sep 18 19:24:44.204083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:44.204048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:45.978236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:45.978203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:46.058009 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:46.057979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:46.090401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:46.090368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz" event={"ID":"395caf8f-4cd7-4c55-bed4-3219dcf1ee3c","Type":"ContainerStarted","Data":"808c1612e4cfc05a67f30602f4403e15f440ae021fc090967d9b8f2ff418f262"}
Sep 18 19:24:46.090563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:46.090471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:46.120750 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:46.120704    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz" podStartSLOduration=1.667504055 podStartE2EDuration="3.120689817s" totalImagesPullingTime="1.453185762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:46.117530922 +0000 UTC m=+646.904811327" watchObservedRunningTime="2026-09-18 19:24:46.120689817 +0000 UTC m=+646.907970220"
Sep 18 19:24:46.578075 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:46.578042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:47.076369 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:47.076326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:47.586088 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:47.586056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:47.831460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:47.831432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:51.556375 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:51.556340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:51.655094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:51.655064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:51.963114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:51.963015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:52.119733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:52.119678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:55.086754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:55.086724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wn4r4"
Sep 18 19:24:55.598060 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:55.598030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:55.754073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:55.754041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:57.097305 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:57.097276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8ndkz"
Sep 18 19:24:57.099458 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:57.099435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:57.217727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:57.217694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:58.206407 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:24:58.206376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:03.622410 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:03.622378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:03.735643 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:03.735615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:03.868945 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:03.868906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:10.666740 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:10.666712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:10.740224 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:10.740196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:14.302029 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:14.301996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:14.448512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:14.448483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:17.393567 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:17.393534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:17.528098 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:17.528068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:20.892193 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:20.892151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:21.019111 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:21.019078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:23.398363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:23.398331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:23.552018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:23.551985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:23.722092 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:23.722012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:42.043455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.043416    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:42.046439 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.046408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:42.047820 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.047804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.052520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.052501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:25:42.052645 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.052626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlhpm\""
Sep 18 19:25:42.060496 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.060473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:42.134941 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.134910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:42.149468 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.149426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:42.152547 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.152521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.152703 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.152683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rspwr\" (UniqueName: \"kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.219790 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.219759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:42.253734 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.253699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rspwr\" (UniqueName: \"kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.253908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.253751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.254417 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.254393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.262707 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.262685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rspwr\" (UniqueName: \"kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr\") pod \"limitador-limitador-8588749977-p9jwn\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") " pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.357566 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.357521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:42.632715 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.632690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:42.634614 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:25:42.634556    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3187f95c_af8d_4487_b556_ba77fa255a69.slice/crio-f45eaca2b7348fff3fdb7391cdc76733c357da7fa4f0676f532a8747b3de6aa3 WatchSource:0}: Error finding container f45eaca2b7348fff3fdb7391cdc76733c357da7fa4f0676f532a8747b3de6aa3: Status 404 returned error can't find the container with id f45eaca2b7348fff3fdb7391cdc76733c357da7fa4f0676f532a8747b3de6aa3
Sep 18 19:25:42.893454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:42.893341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:43.021776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:43.021744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:43.364538 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:43.364470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" event={"ID":"3187f95c-af8d-4487-b556-ba77fa255a69","Type":"ContainerStarted","Data":"f45eaca2b7348fff3fdb7391cdc76733c357da7fa4f0676f532a8747b3de6aa3"}
Sep 18 19:25:44.371429 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:44.371390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" event={"ID":"3187f95c-af8d-4487-b556-ba77fa255a69","Type":"ContainerStarted","Data":"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"}
Sep 18 19:25:44.371913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:44.371451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:44.392804 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:44.392757    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" podStartSLOduration=1.073744232 podStartE2EDuration="2.392742086s" totalImagesPullingTime="1.318997854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:44.390714624 +0000 UTC m=+705.177995029" watchObservedRunningTime="2026-09-18 19:25:44.392742086 +0000 UTC m=+705.180022525"
Sep 18 19:25:55.377304 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:55.377273    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:56.366445 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.366405    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:56.366679 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.366628    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" podUID="3187f95c-af8d-4487-b556-ba77fa255a69" containerName="limitador" containerID="cri-o://d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c" gracePeriod=30
Sep 18 19:25:56.904624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.904598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:56.984303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.984226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rspwr\" (UniqueName: \"kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr\") pod \"3187f95c-af8d-4487-b556-ba77fa255a69\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") "
Sep 18 19:25:56.984435 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.984412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file\") pod \"3187f95c-af8d-4487-b556-ba77fa255a69\" (UID: \"3187f95c-af8d-4487-b556-ba77fa255a69\") "
Sep 18 19:25:56.984780 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.984750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file" (OuterVolumeSpecName: "config-file") pod "3187f95c-af8d-4487-b556-ba77fa255a69" (UID: "3187f95c-af8d-4487-b556-ba77fa255a69"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:25:56.986314 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:56.986295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr" (OuterVolumeSpecName: "kube-api-access-rspwr") pod "3187f95c-af8d-4487-b556-ba77fa255a69" (UID: "3187f95c-af8d-4487-b556-ba77fa255a69"). InnerVolumeSpecName "kube-api-access-rspwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:25:57.085957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.085923    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rspwr\" (UniqueName: \"kubernetes.io/projected/3187f95c-af8d-4487-b556-ba77fa255a69-kube-api-access-rspwr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:25:57.085957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.085949    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3187f95c-af8d-4487-b556-ba77fa255a69-config-file\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:25:57.435657 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.435621    2580 generic.go:356] "Generic (PLEG): container finished" podID="3187f95c-af8d-4487-b556-ba77fa255a69" containerID="d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c" exitCode=0
Sep 18 19:25:57.435816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.435665    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn"
Sep 18 19:25:57.435816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.435705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" event={"ID":"3187f95c-af8d-4487-b556-ba77fa255a69","Type":"ContainerDied","Data":"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"}
Sep 18 19:25:57.435816 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.435798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p9jwn" event={"ID":"3187f95c-af8d-4487-b556-ba77fa255a69","Type":"ContainerDied","Data":"f45eaca2b7348fff3fdb7391cdc76733c357da7fa4f0676f532a8747b3de6aa3"}
Sep 18 19:25:57.435965 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.435827    2580 scope.go:117] "RemoveContainer" containerID="d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"
Sep 18 19:25:57.444141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.444122    2580 scope.go:117] "RemoveContainer" containerID="d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"
Sep 18 19:25:57.444398 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:25:57.444382    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c\": container with ID starting with d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c not found: ID does not exist" containerID="d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"
Sep 18 19:25:57.444440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.444405    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c"} err="failed to get container status \"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c\": rpc error: code = NotFound desc = could not find container \"d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c\": container with ID starting with d67ea58d52004df5cb221be60463aebb9cc2927b84451277fb2c9160f5d6ab1c not found: ID does not exist"
Sep 18 19:25:57.456308 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.456282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:57.463169 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:57.463147    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p9jwn"]
Sep 18 19:25:59.293742 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:25:59.293707    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3187f95c-af8d-4487-b556-ba77fa255a69" path="/var/lib/kubelet/pods/3187f95c-af8d-4487-b556-ba77fa255a69/volumes"
Sep 18 19:26:02.389633 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:02.389577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:03.780309 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:03.780270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:11.218997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:11.218958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:11.386228 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:11.386193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:28.572559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.572526    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"]
Sep 18 19:26:28.573073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.573055    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3187f95c-af8d-4487-b556-ba77fa255a69" containerName="limitador"
Sep 18 19:26:28.573118 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.573077    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3187f95c-af8d-4487-b556-ba77fa255a69" containerName="limitador"
Sep 18 19:26:28.573202 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.573191    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3187f95c-af8d-4487-b556-ba77fa255a69" containerName="limitador"
Sep 18 19:26:28.576401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.576379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.593357 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.593327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"]
Sep 18 19:26:28.661665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tbn\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-kube-api-access-n5tbn\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661899 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661899 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661899 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661899 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.661899 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.661874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tbn\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-kube-api-access-n5tbn\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.762702 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.762643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.763713 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.763682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.765696 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.765638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.765696 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.765673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.767172 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.767122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.767335 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.767291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.799957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.799924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.800135 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.800043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tbn\" (UniqueName: \"kubernetes.io/projected/fdd752a6-29ef-42f8-9930-9d1fcdaa389e-kube-api-access-n5tbn\") pod \"istiod-openshift-gateway-5677ccb49b-bgpc8\" (UID: \"fdd752a6-29ef-42f8-9930-9d1fcdaa389e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:28.885277 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:28.885185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:29.191889 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.191865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"]
Sep 18 19:26:29.193691 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:26:29.193667    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd752a6_29ef_42f8_9930_9d1fcdaa389e.slice/crio-b42f44ea91aa65058305b201a1e5ab0570d6aa988bb333a5c277a2bccb4ba189 WatchSource:0}: Error finding container b42f44ea91aa65058305b201a1e5ab0570d6aa988bb333a5c277a2bccb4ba189: Status 404 returned error can't find the container with id b42f44ea91aa65058305b201a1e5ab0570d6aa988bb333a5c277a2bccb4ba189
Sep 18 19:26:29.196258 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.196222    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:26:29.196342 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.196281    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:26:29.583771 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.583727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8" event={"ID":"fdd752a6-29ef-42f8-9930-9d1fcdaa389e","Type":"ContainerStarted","Data":"671657bb9ca04a01ab718e05bb1090f6a0c930a473761ec52360cbc7f1b78c32"}
Sep 18 19:26:29.584175 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.583794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8" event={"ID":"fdd752a6-29ef-42f8-9930-9d1fcdaa389e","Type":"ContainerStarted","Data":"b42f44ea91aa65058305b201a1e5ab0570d6aa988bb333a5c277a2bccb4ba189"}
Sep 18 19:26:29.584175 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.583836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:29.627164 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:29.627109    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8" podStartSLOduration=1.6270923929999999 podStartE2EDuration="1.627092393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:29.624043667 +0000 UTC m=+750.411324071" watchObservedRunningTime="2026-09-18 19:26:29.627092393 +0000 UTC m=+750.414372797"
Sep 18 19:26:30.591073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:30.591036    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-bgpc8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:26:30.591443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:30.591096    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8" podUID="fdd752a6-29ef-42f8-9930-9d1fcdaa389e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:33.590413 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:33.590382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bgpc8"
Sep 18 19:26:33.680037 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:33.680004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:26:33.680230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:33.680209    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerName="discovery" containerID="cri-o://a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8" gracePeriod=30
Sep 18 19:26:33.930956 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:33.930930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:26:34.005366 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2g42\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005517    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token\") pod \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\" (UID: \"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc\") "
Sep 18 19:26:34.005992 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.005943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:26:34.007884 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.007861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs" (OuterVolumeSpecName: "local-certs") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:26:34.007987 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.007880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42" (OuterVolumeSpecName: "kube-api-access-q2g42") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "kube-api-access-q2g42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:26:34.008064 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.008040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts" (OuterVolumeSpecName: "cacerts") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:26:34.008117 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.008060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:26:34.008325 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.008154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:26:34.008325 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.008156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" (UID: "99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107037    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-local-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107089    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2g42\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-kube-api-access-q2g42\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107106    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-cacerts\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107120    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-dns-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107134    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-kubeconfig\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107152    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-csr-ca-configmap\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.111672 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.107166    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:26:34.609058 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.609020    2580 generic.go:356] "Generic (PLEG): container finished" podID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerID="a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8" exitCode=0
Sep 18 19:26:34.609455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.609077    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"
Sep 18 19:26:34.609455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.609105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" event={"ID":"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc","Type":"ContainerDied","Data":"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"}
Sep 18 19:26:34.609455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.609160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6" event={"ID":"99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc","Type":"ContainerDied","Data":"d367a70345622a1f0a11fc21ef5e34a52827e77bd642c0b508a34255879dd198"}
Sep 18 19:26:34.609455 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.609181    2580 scope.go:117] "RemoveContainer" containerID="a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"
Sep 18 19:26:34.618542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.618522    2580 scope.go:117] "RemoveContainer" containerID="a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"
Sep 18 19:26:34.618847 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:26:34.618831    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8\": container with ID starting with a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8 not found: ID does not exist" containerID="a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"
Sep 18 19:26:34.618900 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.618855    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8"} err="failed to get container status \"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8\": rpc error: code = NotFound desc = could not find container \"a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8\": container with ID starting with a55d05c10fd707275aa504ff6a12d8d19c708cdcf5532055dca533d167f4d0e8 not found: ID does not exist"
Sep 18 19:26:34.641180 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.641148    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:26:34.647657 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:34.647632    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-4x4m6"]
Sep 18 19:26:35.294100 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:35.294073    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" path="/var/lib/kubelet/pods/99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc/volumes"
Sep 18 19:26:44.297242 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.297209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:44.383602 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.383555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:44.383959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.383937    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:26:44.384355 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.384342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerName="discovery"
Sep 18 19:26:44.384392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.384358    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerName="discovery"
Sep 18 19:26:44.384426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.384419    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f2e2a1-6401-4972-af6c-ea2d1fb3a6cc" containerName="discovery"
Sep 18 19:26:44.388754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.388736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.392633 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.392607    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:26:44.394272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.394252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fl46d\""
Sep 18 19:26:44.394390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.394308    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 19:26:44.394390 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.394378    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:26:44.399714 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.399694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:26:44.428215 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.428184    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-65474cf8d5-6msh7"]
Sep 18 19:26:44.432055 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.432033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.435074 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.435054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-72pdw\""
Sep 18 19:26:44.435306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.435290    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 19:26:44.446563 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.446542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-65474cf8d5-6msh7"]
Sep 18 19:26:44.454861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.454841    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qqcnr"]
Sep 18 19:26:44.458194 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.458173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.462086 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.462060    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:26:44.462183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.462145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f2gwx\""
Sep 18 19:26:44.470724 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.470701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qqcnr"]
Sep 18 19:26:44.495255 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-data\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.495377 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhndx\" (UniqueName: \"kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.495377 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.495377 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.495520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfdz\" (UniqueName: \"kubernetes.io/projected/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-kube-api-access-2pfdz\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.495520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.495456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gq7\" (UniqueName: \"kubernetes.io/projected/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-kube-api-access-98gq7\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.501236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.501216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:44.580146 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.580071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:44.596173 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.596287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.596287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfdz\" (UniqueName: \"kubernetes.io/projected/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-kube-api-access-2pfdz\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.596287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gq7\" (UniqueName: \"kubernetes.io/projected/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-kube-api-access-98gq7\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.596287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-data\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.596481 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhndx\" (UniqueName: \"kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.596481 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:26:44.596308    2580 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 19:26:44.596481 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:26:44.596375    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert podName:ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:26:45.096354456 +0000 UTC m=+765.883634842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert") pod "llmisvc-controller-manager-65474cf8d5-6msh7" (UID: "ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9") : secret "llmisvc-webhook-server-cert" not found
Sep 18 19:26:44.596733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.596717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-data\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.598725 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.598707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.607738 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.607717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhndx\" (UniqueName: \"kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx\") pod \"kserve-controller-manager-5ccfd4fdb4-s6kjc\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.607898 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.607880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfdz\" (UniqueName: \"kubernetes.io/projected/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-kube-api-access-2pfdz\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:44.608142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.608127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gq7\" (UniqueName: \"kubernetes.io/projected/2270b2cf-aaee-4ef4-b0e4-4bc15e576c33-kube-api-access-98gq7\") pod \"seaweedfs-64568bcd49-qqcnr\" (UID: \"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33\") " pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.700094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.700062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:44.768567 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.768539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:44.980145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.980122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:26:44.981940 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:26:44.981917    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ed204e_f715_4b27_8b52_3a16c7486f10.slice/crio-f596d4e614d732f440f5b221962b00cad98ec0b1310222f178c3048d000bed7e WatchSource:0}: Error finding container f596d4e614d732f440f5b221962b00cad98ec0b1310222f178c3048d000bed7e: Status 404 returned error can't find the container with id f596d4e614d732f440f5b221962b00cad98ec0b1310222f178c3048d000bed7e
Sep 18 19:26:44.986346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:44.986320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:45.050372 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.050348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qqcnr"]
Sep 18 19:26:45.052446 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:26:45.052423    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2270b2cf_aaee_4ef4_b0e4_4bc15e576c33.slice/crio-300cc88333983e7428c017dd0f0e7d579a3626fa0ed51c35efcbf0235b4f525f WatchSource:0}: Error finding container 300cc88333983e7428c017dd0f0e7d579a3626fa0ed51c35efcbf0235b4f525f: Status 404 returned error can't find the container with id 300cc88333983e7428c017dd0f0e7d579a3626fa0ed51c35efcbf0235b4f525f
Sep 18 19:26:45.100959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.100927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:45.102682 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.102653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:45.103450 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.103429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9-cert\") pod \"llmisvc-controller-manager-65474cf8d5-6msh7\" (UID: \"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9\") " pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:45.203229 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.203191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:45.310990 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.310957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:45.341796 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.341769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:45.641067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.640364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-65474cf8d5-6msh7"]
Sep 18 19:26:45.647041 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:26:45.647010    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee309e4c_2eb0_4bdd_9483_ec5a7fbd22e9.slice/crio-90a8deb0dc4e8b9847a2124443a3588dca13a7c82b1f67ff30ca3f44ef8baad1 WatchSource:0}: Error finding container 90a8deb0dc4e8b9847a2124443a3588dca13a7c82b1f67ff30ca3f44ef8baad1: Status 404 returned error can't find the container with id 90a8deb0dc4e8b9847a2124443a3588dca13a7c82b1f67ff30ca3f44ef8baad1
Sep 18 19:26:45.666815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.666779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" event={"ID":"e2ed204e-f715-4b27-8b52-3a16c7486f10","Type":"ContainerStarted","Data":"f596d4e614d732f440f5b221962b00cad98ec0b1310222f178c3048d000bed7e"}
Sep 18 19:26:45.668659 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.668626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7" event={"ID":"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9","Type":"ContainerStarted","Data":"90a8deb0dc4e8b9847a2124443a3588dca13a7c82b1f67ff30ca3f44ef8baad1"}
Sep 18 19:26:45.671652 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:45.671595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qqcnr" event={"ID":"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33","Type":"ContainerStarted","Data":"300cc88333983e7428c017dd0f0e7d579a3626fa0ed51c35efcbf0235b4f525f"}
Sep 18 19:26:49.709311 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.709280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qqcnr" event={"ID":"2270b2cf-aaee-4ef4-b0e4-4bc15e576c33","Type":"ContainerStarted","Data":"72a7415051aec145d81828619c3c7efa3d7a68aebd41c80eba1512a8d7eb36b2"}
Sep 18 19:26:49.709901 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.709351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:26:49.711114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.711086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" event={"ID":"e2ed204e-f715-4b27-8b52-3a16c7486f10","Type":"ContainerStarted","Data":"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"}
Sep 18 19:26:49.711237 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.711220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:26:49.733949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.733894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qqcnr" podStartSLOduration=1.157439142 podStartE2EDuration="5.733877773s" totalImagesPullingTime="4.576438631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:49.733134716 +0000 UTC m=+770.520415122" watchObservedRunningTime="2026-09-18 19:26:49.733877773 +0000 UTC m=+770.521158188"
Sep 18 19:26:49.756026 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:49.755983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" podStartSLOduration=1.239983301 podStartE2EDuration="5.755967916s" totalImagesPullingTime="4.515984615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:49.753996634 +0000 UTC m=+770.541277041" watchObservedRunningTime="2026-09-18 19:26:49.755967916 +0000 UTC m=+770.543248366"
Sep 18 19:26:50.717789 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:50.717733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7" event={"ID":"ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9","Type":"ContainerStarted","Data":"7574f956c17ff6db36c599d2c655298aceabc19eb440f93cb802c34ea73c5602"}
Sep 18 19:26:50.718246 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:50.718036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:26:50.737353 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:50.737306    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7" podStartSLOduration=2.849289551 podStartE2EDuration="6.737289927s" totalImagesPullingTime="3.888000376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:50.735964747 +0000 UTC m=+771.523245152" watchObservedRunningTime="2026-09-18 19:26:50.737289927 +0000 UTC m=+771.524570332"
Sep 18 19:26:55.720269 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:26:55.720236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qqcnr"
Sep 18 19:27:20.722530 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:20.722500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:27:21.437335 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:21.437299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:21.725552 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:21.725477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-65474cf8d5-6msh7"
Sep 18 19:27:30.097689 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.097656    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:27:30.098209 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.097844    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" podUID="e2ed204e-f715-4b27-8b52-3a16c7486f10" containerName="manager" containerID="cri-o://704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d" gracePeriod=10
Sep 18 19:27:30.124515 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.124488    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"]
Sep 18 19:27:30.127022 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.127007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.136831 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.136796    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"]
Sep 18 19:27:30.197769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.197744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00f7e83b-ee05-4475-9218-3be06c564f7e-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.197882 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.197800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8855\" (UniqueName: \"kubernetes.io/projected/00f7e83b-ee05-4475-9218-3be06c564f7e-kube-api-access-x8855\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.298427 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.298394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00f7e83b-ee05-4475-9218-3be06c564f7e-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.298604 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.298445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8855\" (UniqueName: \"kubernetes.io/projected/00f7e83b-ee05-4475-9218-3be06c564f7e-kube-api-access-x8855\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.300713 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.300689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00f7e83b-ee05-4475-9218-3be06c564f7e-cert\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.307914 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.307892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8855\" (UniqueName: \"kubernetes.io/projected/00f7e83b-ee05-4475-9218-3be06c564f7e-kube-api-access-x8855\") pod \"kserve-controller-manager-5ccfd4fdb4-p2p95\" (UID: \"00f7e83b-ee05-4475-9218-3be06c564f7e\") " pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.334304 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.334284    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:27:30.399737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.399573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert\") pod \"e2ed204e-f715-4b27-8b52-3a16c7486f10\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") "
Sep 18 19:27:30.399737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.399621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhndx\" (UniqueName: \"kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx\") pod \"e2ed204e-f715-4b27-8b52-3a16c7486f10\" (UID: \"e2ed204e-f715-4b27-8b52-3a16c7486f10\") "
Sep 18 19:27:30.401657 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.401631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx" (OuterVolumeSpecName: "kube-api-access-qhndx") pod "e2ed204e-f715-4b27-8b52-3a16c7486f10" (UID: "e2ed204e-f715-4b27-8b52-3a16c7486f10"). InnerVolumeSpecName "kube-api-access-qhndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:27:30.401762 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.401681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert" (OuterVolumeSpecName: "cert") pod "e2ed204e-f715-4b27-8b52-3a16c7486f10" (UID: "e2ed204e-f715-4b27-8b52-3a16c7486f10"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:30.467376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.467346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:30.500872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.500839    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhndx\" (UniqueName: \"kubernetes.io/projected/e2ed204e-f715-4b27-8b52-3a16c7486f10-kube-api-access-qhndx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:27:30.500872 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.500873    2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ed204e-f715-4b27-8b52-3a16c7486f10-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:27:30.742454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.742375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"]
Sep 18 19:27:30.745332 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:27:30.745303    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7e83b_ee05_4475_9218_3be06c564f7e.slice/crio-38544f687854682072631ab56a6afa2df33819c0052481e088796f81d06e5d9b WatchSource:0}: Error finding container 38544f687854682072631ab56a6afa2df33819c0052481e088796f81d06e5d9b: Status 404 returned error can't find the container with id 38544f687854682072631ab56a6afa2df33819c0052481e088796f81d06e5d9b
Sep 18 19:27:30.746720 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.746703    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:27:30.904991 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.904953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95" event={"ID":"00f7e83b-ee05-4475-9218-3be06c564f7e","Type":"ContainerStarted","Data":"38544f687854682072631ab56a6afa2df33819c0052481e088796f81d06e5d9b"}
Sep 18 19:27:30.906252 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.906225    2580 generic.go:356] "Generic (PLEG): container finished" podID="e2ed204e-f715-4b27-8b52-3a16c7486f10" containerID="704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d" exitCode=0
Sep 18 19:27:30.906380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.906275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" event={"ID":"e2ed204e-f715-4b27-8b52-3a16c7486f10","Type":"ContainerDied","Data":"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"}
Sep 18 19:27:30.906380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.906279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"
Sep 18 19:27:30.906380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.906314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc" event={"ID":"e2ed204e-f715-4b27-8b52-3a16c7486f10","Type":"ContainerDied","Data":"f596d4e614d732f440f5b221962b00cad98ec0b1310222f178c3048d000bed7e"}
Sep 18 19:27:30.906380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.906336    2580 scope.go:117] "RemoveContainer" containerID="704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"
Sep 18 19:27:30.914554 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.914538    2580 scope.go:117] "RemoveContainer" containerID="704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"
Sep 18 19:27:30.914830 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:27:30.914812    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d\": container with ID starting with 704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d not found: ID does not exist" containerID="704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"
Sep 18 19:27:30.914897 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.914838    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d"} err="failed to get container status \"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d\": rpc error: code = NotFound desc = could not find container \"704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d\": container with ID starting with 704e062134ce735e27f87d6c61b0a432ac2e9aac9241b47cf3a419f357542e4d not found: ID does not exist"
Sep 18 19:27:30.928376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.928353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:27:30.930673 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:30.930653    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5ccfd4fdb4-s6kjc"]
Sep 18 19:27:31.294350 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:31.294323    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ed204e-f715-4b27-8b52-3a16c7486f10" path="/var/lib/kubelet/pods/e2ed204e-f715-4b27-8b52-3a16c7486f10/volumes"
Sep 18 19:27:31.913236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:31.913206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95" event={"ID":"00f7e83b-ee05-4475-9218-3be06c564f7e","Type":"ContainerStarted","Data":"fbef868f47a062b199a1c4aeb1c466135577a2b31207f6cdd0e7ca760369c94e"}
Sep 18 19:27:31.913397 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:31.913274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:27:31.948532 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:31.948486    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95" podStartSLOduration=1.544102235 podStartE2EDuration="1.948472381s" totalImagesPullingTime="404.370146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:31.947943435 +0000 UTC m=+812.735223837" watchObservedRunningTime="2026-09-18 19:27:31.948472381 +0000 UTC m=+812.735752783"
Sep 18 19:27:57.592252 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:57.592217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:57.693546 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:27:57.693515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:02.921974 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:02.921946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5ccfd4fdb4-p2p95"
Sep 18 19:28:04.361346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.361312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:04.413067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.413035    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-29w5w"]
Sep 18 19:28:04.413491 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.413473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ed204e-f715-4b27-8b52-3a16c7486f10" containerName="manager"
Sep 18 19:28:04.413491 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.413492    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ed204e-f715-4b27-8b52-3a16c7486f10" containerName="manager"
Sep 18 19:28:04.413677 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.413599    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ed204e-f715-4b27-8b52-3a16c7486f10" containerName="manager"
Sep 18 19:28:04.416380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.416361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.422907 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.422886    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 19:28:04.423025 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.422887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z7wnh\""
Sep 18 19:28:04.424140 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.424119    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-24z7c"]
Sep 18 19:28:04.426890 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.426876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.429945 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.429922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:28:04.430059 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.429969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5zrqn\""
Sep 18 19:28:04.430059 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.429976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:28:04.430554 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.430534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-29w5w"]
Sep 18 19:28:04.444206 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.444182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-24z7c"]
Sep 18 19:28:04.468235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.468208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:04.567978 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.567945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:04.588298 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.588270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7972b76-3801-45cd-89fc-8a932555f309-tls-certs\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.588454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.588303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-cert\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.588454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.588323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7gx\" (UniqueName: \"kubernetes.io/projected/41294152-b30a-46f7-b1a0-8b13ede112b6-kube-api-access-5x7gx\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.588454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.588343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p255h\" (UniqueName: \"kubernetes.io/projected/a7972b76-3801-45cd-89fc-8a932555f309-kube-api-access-p255h\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.588454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.588388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.668195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.668121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:04.689250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.689223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7972b76-3801-45cd-89fc-8a932555f309-tls-certs\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.689368 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.689256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-cert\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.689368 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.689277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7gx\" (UniqueName: \"kubernetes.io/projected/41294152-b30a-46f7-b1a0-8b13ede112b6-kube-api-access-5x7gx\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.689368 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.689294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p255h\" (UniqueName: \"kubernetes.io/projected/a7972b76-3801-45cd-89fc-8a932555f309-kube-api-access-p255h\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.689495 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.689394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.689567 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:28:04.689549    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 19:28:04.689677 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:28:04.689653    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs podName:41294152-b30a-46f7-b1a0-8b13ede112b6 nodeName:}" failed. No retries permitted until 2026-09-18 19:28:05.189629402 +0000 UTC m=+845.976909790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs") pod "odh-model-controller-7b966d7984-24z7c" (UID: "41294152-b30a-46f7-b1a0-8b13ede112b6") : secret "odh-model-controller-metrics-tls" not found
Sep 18 19:28:04.691811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.691785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-cert\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.691811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.691797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7972b76-3801-45cd-89fc-8a932555f309-tls-certs\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.702230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.702211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p255h\" (UniqueName: \"kubernetes.io/projected/a7972b76-3801-45cd-89fc-8a932555f309-kube-api-access-p255h\") pod \"model-serving-api-7b954ffc95-29w5w\" (UID: \"a7972b76-3801-45cd-89fc-8a932555f309\") " pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:04.702454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.702437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7gx\" (UniqueName: \"kubernetes.io/projected/41294152-b30a-46f7-b1a0-8b13ede112b6-kube-api-access-5x7gx\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:04.727462 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:04.727439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:05.019143 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.019113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-29w5w"]
Sep 18 19:28:05.020244 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:28:05.020224    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7972b76_3801_45cd_89fc_8a932555f309.slice/crio-66a4f116628e1537a3c301760b5751d542599c776fc02522a9bc182fe3053369 WatchSource:0}: Error finding container 66a4f116628e1537a3c301760b5751d542599c776fc02522a9bc182fe3053369: Status 404 returned error can't find the container with id 66a4f116628e1537a3c301760b5751d542599c776fc02522a9bc182fe3053369
Sep 18 19:28:05.023763 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.023740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:05.068998 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.068968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-29w5w" event={"ID":"a7972b76-3801-45cd-89fc-8a932555f309","Type":"ContainerStarted","Data":"66a4f116628e1537a3c301760b5751d542599c776fc02522a9bc182fe3053369"}
Sep 18 19:28:05.194514 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.194480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:05.196815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.196790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41294152-b30a-46f7-b1a0-8b13ede112b6-metrics-certs\") pod \"odh-model-controller-7b966d7984-24z7c\" (UID: \"41294152-b30a-46f7-b1a0-8b13ede112b6\") " pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:05.337147 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.337118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:05.615070 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.615038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-24z7c"]
Sep 18 19:28:05.617455 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:28:05.617418    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41294152_b30a_46f7_b1a0_8b13ede112b6.slice/crio-a5e522b5d746d967e1b42373c6c17732790e23e0eb3d1c086a57310d60f481b6 WatchSource:0}: Error finding container a5e522b5d746d967e1b42373c6c17732790e23e0eb3d1c086a57310d60f481b6: Status 404 returned error can't find the container with id a5e522b5d746d967e1b42373c6c17732790e23e0eb3d1c086a57310d60f481b6
Sep 18 19:28:05.769911 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.769880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:05.853405 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:05.853371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:06.076436 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:06.076398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-24z7c" event={"ID":"41294152-b30a-46f7-b1a0-8b13ede112b6","Type":"ContainerStarted","Data":"a5e522b5d746d967e1b42373c6c17732790e23e0eb3d1c086a57310d60f481b6"}
Sep 18 19:28:07.085352 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:07.085305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-29w5w" event={"ID":"a7972b76-3801-45cd-89fc-8a932555f309","Type":"ContainerStarted","Data":"42689680a0d5550c34f2e982a4371cf73b3cc86f93d598494bb4bce19799844d"}
Sep 18 19:28:07.085850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:07.085445    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:07.108904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:07.108857    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-29w5w" podStartSLOduration=1.5366655009999999 podStartE2EDuration="3.10883622s" totalImagesPullingTime="1.572170719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:28:07.106776199 +0000 UTC m=+847.894056602" watchObservedRunningTime="2026-09-18 19:28:07.10883622 +0000 UTC m=+847.896116624"
Sep 18 19:28:09.095469 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:09.095432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-24z7c" event={"ID":"41294152-b30a-46f7-b1a0-8b13ede112b6","Type":"ContainerStarted","Data":"e355a46b379ee53bc05f8805e4ce9a7e5f27f70d0f1755a5c1629b5a564fc3b4"}
Sep 18 19:28:09.095827 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:09.095540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:09.119028 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:09.118983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-24z7c" podStartSLOduration=2.655680275 podStartE2EDuration="5.118967992s" totalImagesPullingTime="2.463287717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:28:09.118717614 +0000 UTC m=+849.905998017" watchObservedRunningTime="2026-09-18 19:28:09.118967992 +0000 UTC m=+849.906248433"
Sep 18 19:28:18.093684 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:18.093650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-29w5w"
Sep 18 19:28:20.103406 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:20.103374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-24z7c"
Sep 18 19:28:53.310100 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:53.310020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:53.390533 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:28:53.390503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:29:01.237665 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:01.237639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:29:01.238643 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:01.238623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:29:14.564202 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:14.564145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:29:40.085772 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:40.085731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:29:40.183131 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:40.183087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:29:53.706499 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:53.706462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:29:53.799229 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:29:53.799201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:03.883753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:03.883704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:32.178157 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.178107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:32.472248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.472181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:30:32.475799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.475772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.483195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.482775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-bgtp6\""
Sep 18 19:30:32.483195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.482951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 19:30:32.483195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.483118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-24jtl\""
Sep 18 19:30:32.483195 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.483145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 19:30:32.483477 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.483199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 19:30:32.493497 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.493476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:30:32.508694 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.508666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:32.563926 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.563897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.564051 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.563935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jmd\" (UniqueName: \"kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.664921 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.664889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jmd\" (UniqueName: \"kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.665094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.665001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.667812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.667784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.678213 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.678191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jmd\" (UniqueName: \"kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:32.786688 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:32.786610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:33.062052 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:33.062026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:30:33.064266 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:30:33.064245    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc007269a_6b7b_4bf1_a7e1_5bd784904c31.slice/crio-581bc36233178b4b00b8d6927904bf9b41c72dc497788945aff4112febff15a0 WatchSource:0}: Error finding container 581bc36233178b4b00b8d6927904bf9b41c72dc497788945aff4112febff15a0: Status 404 returned error can't find the container with id 581bc36233178b4b00b8d6927904bf9b41c72dc497788945aff4112febff15a0
Sep 18 19:30:33.805719 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:33.805681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" event={"ID":"c007269a-6b7b-4bf1-a7e1-5bd784904c31","Type":"ContainerStarted","Data":"581bc36233178b4b00b8d6927904bf9b41c72dc497788945aff4112febff15a0"}
Sep 18 19:30:34.153345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:34.153265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:35.819699 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:35.819666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" event={"ID":"c007269a-6b7b-4bf1-a7e1-5bd784904c31","Type":"ContainerStarted","Data":"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"}
Sep 18 19:30:35.820087 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:35.819793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:30:35.840034 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:35.839988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podStartSLOduration=1.2561864169999999 podStartE2EDuration="3.83997405s" totalImagesPullingTime="2.583787633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:30:35.838001551 +0000 UTC m=+996.625281954" watchObservedRunningTime="2026-09-18 19:30:35.83997405 +0000 UTC m=+996.627254452"
Sep 18 19:30:40.070684 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:30:40.070648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:06.827175 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:06.827146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:31:08.017947 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:08.017909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:08.156634 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:08.156598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:31:09.225739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:09.225706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:16.826853 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:16.826817    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:31:23.177831 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:23.177792    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main" containerID="cri-o://4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862" gracePeriod=45
Sep 18 19:31:26.826961 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:26.826921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:31:36.826620 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:36.826567    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:31:36.827027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:36.826698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:31:46.827245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:46.827160    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:31:53.429326 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.429302    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:31:53.463506 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.463433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98jmd\" (UniqueName: \"kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd\") pod \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") "
Sep 18 19:31:53.463666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.463631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs\") pod \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\" (UID: \"c007269a-6b7b-4bf1-a7e1-5bd784904c31\") "
Sep 18 19:31:53.465631 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.465601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c007269a-6b7b-4bf1-a7e1-5bd784904c31" (UID: "c007269a-6b7b-4bf1-a7e1-5bd784904c31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:53.465731 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.465636    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd" (OuterVolumeSpecName: "kube-api-access-98jmd") pod "c007269a-6b7b-4bf1-a7e1-5bd784904c31" (UID: "c007269a-6b7b-4bf1-a7e1-5bd784904c31"). InnerVolumeSpecName "kube-api-access-98jmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:53.565186 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.565159    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jmd\" (UniqueName: \"kubernetes.io/projected/c007269a-6b7b-4bf1-a7e1-5bd784904c31-kube-api-access-98jmd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:31:53.565186 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:53.565183    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c007269a-6b7b-4bf1-a7e1-5bd784904c31-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:31:54.210413 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.210376    2580 generic.go:356] "Generic (PLEG): container finished" podID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerID="4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862" exitCode=0
Sep 18 19:31:54.210413 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.210416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" event={"ID":"c007269a-6b7b-4bf1-a7e1-5bd784904c31","Type":"ContainerDied","Data":"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"}
Sep 18 19:31:54.210794 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.210437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"
Sep 18 19:31:54.210794 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.210482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl" event={"ID":"c007269a-6b7b-4bf1-a7e1-5bd784904c31","Type":"ContainerDied","Data":"581bc36233178b4b00b8d6927904bf9b41c72dc497788945aff4112febff15a0"}
Sep 18 19:31:54.210794 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.210503    2580 scope.go:117] "RemoveContainer" containerID="4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"
Sep 18 19:31:54.219782 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.219764    2580 scope.go:117] "RemoveContainer" containerID="4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"
Sep 18 19:31:54.220049 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:31:54.220029    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862\": container with ID starting with 4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862 not found: ID does not exist" containerID="4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"
Sep 18 19:31:54.220092 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.220057    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862"} err="failed to get container status \"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862\": rpc error: code = NotFound desc = could not find container \"4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862\": container with ID starting with 4355b51fd191df5f69e5db13906c8dccd5bd97f335a64557aa99b72fe8df7862 not found: ID does not exist"
Sep 18 19:31:54.240212 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.240191    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:31:54.247516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:54.247497    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6tzgl"]
Sep 18 19:31:55.293781 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:55.293747    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" path="/var/lib/kubelet/pods/c007269a-6b7b-4bf1-a7e1-5bd784904c31/volumes"
Sep 18 19:31:58.122154 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:58.122124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:58.232069 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:58.232030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:59.538693 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:59.538636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:59.635128 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:59.635089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:59.741131 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:31:59.741095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:05.272686 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:05.272650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:06.798130 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:06.798088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:06.913959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:06.913921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:07.277361 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:07.277272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:07.398870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:07.398835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:11.561747 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:11.561718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:11.656481 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:11.656452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:24.379697 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.379658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:24.755783 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.755701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:32:24.756169 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.756150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main"
Sep 18 19:32:24.756256 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.756172    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main"
Sep 18 19:32:24.756308 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.756276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c007269a-6b7b-4bf1-a7e1-5bd784904c31" containerName="main"
Sep 18 19:32:24.759333 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.759312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.774096 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.774074    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 19:32:24.774235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.774092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 19:32:24.774235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.774093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-wszkc\""
Sep 18 19:32:24.774235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.774107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 19:32:24.789142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.789109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tzmj8\""
Sep 18 19:32:24.791225 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.791197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:32:24.842809 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.842774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.842957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.842852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7v6l\" (UniqueName: \"kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.898850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.898816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:24.943632 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.943577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.943794 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.943664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7v6l\" (UniqueName: \"kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.946254 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.946231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:24.955518 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:24.955497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7v6l\" (UniqueName: \"kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:25.069364 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:25.069333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:25.350428 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:25.350406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:32:25.352987 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:32:25.352962    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8cea04_854b_45b0_9f17_2784e5591ae5.slice/crio-9950e820efb379ad425d5923b6bdd14e2d2b624fe0db4c4354d93ea6a4f290f5 WatchSource:0}: Error finding container 9950e820efb379ad425d5923b6bdd14e2d2b624fe0db4c4354d93ea6a4f290f5: Status 404 returned error can't find the container with id 9950e820efb379ad425d5923b6bdd14e2d2b624fe0db4c4354d93ea6a4f290f5
Sep 18 19:32:25.360259 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:25.360081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" event={"ID":"4d8cea04-854b-45b0-9f17-2784e5591ae5","Type":"ContainerStarted","Data":"9950e820efb379ad425d5923b6bdd14e2d2b624fe0db4c4354d93ea6a4f290f5"}
Sep 18 19:32:26.369683 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:26.369649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" event={"ID":"4d8cea04-854b-45b0-9f17-2784e5591ae5","Type":"ContainerStarted","Data":"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"}
Sep 18 19:32:26.370050 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:26.369768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:32:26.397232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:26.397186    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podStartSLOduration=2.39717015 podStartE2EDuration="2.39717015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:26.396117722 +0000 UTC m=+1107.183398131" watchObservedRunningTime="2026-09-18 19:32:26.39717015 +0000 UTC m=+1107.184450555"
Sep 18 19:32:37.339842 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:37.339810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:38.745140 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:38.745106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:57.375654 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:32:57.375622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:33:01.588225 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:01.588190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:03.002874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:03.002829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:03.010160 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:03.010133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:33:03.098338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:03.098308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:07.375562 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:07.375528    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:33:17.376049 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:17.375962    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:33:18.029630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:18.029548    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" containerID="cri-o://8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec" gracePeriod=45
Sep 18 19:33:26.521140 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:26.521105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:26.619257 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:26.619218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:27.376214 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:27.376173    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:33:27.376440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:27.376301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:33:37.375780 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:37.375743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:33:47.375997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:47.375961    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:33:48.271127 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.271100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:33:48.336804 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.336772    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs\") pod \"4d8cea04-854b-45b0-9f17-2784e5591ae5\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") "
Sep 18 19:33:48.337009 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.336854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7v6l\" (UniqueName: \"kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l\") pod \"4d8cea04-854b-45b0-9f17-2784e5591ae5\" (UID: \"4d8cea04-854b-45b0-9f17-2784e5591ae5\") "
Sep 18 19:33:48.338998 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.338975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d8cea04-854b-45b0-9f17-2784e5591ae5" (UID: "4d8cea04-854b-45b0-9f17-2784e5591ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:48.339068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.339024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l" (OuterVolumeSpecName: "kube-api-access-s7v6l") pod "4d8cea04-854b-45b0-9f17-2784e5591ae5" (UID: "4d8cea04-854b-45b0-9f17-2784e5591ae5"). InnerVolumeSpecName "kube-api-access-s7v6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:48.438236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.438205    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8cea04-854b-45b0-9f17-2784e5591ae5-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:33:48.438236 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.438234    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7v6l\" (UniqueName: \"kubernetes.io/projected/4d8cea04-854b-45b0-9f17-2784e5591ae5-kube-api-access-s7v6l\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:33:48.783359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.783274    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerID="8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec" exitCode=0
Sep 18 19:33:48.783359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.783317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"
Sep 18 19:33:48.783359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.783325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" event={"ID":"4d8cea04-854b-45b0-9f17-2784e5591ae5","Type":"ContainerDied","Data":"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"}
Sep 18 19:33:48.783577 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.783374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl" event={"ID":"4d8cea04-854b-45b0-9f17-2784e5591ae5","Type":"ContainerDied","Data":"9950e820efb379ad425d5923b6bdd14e2d2b624fe0db4c4354d93ea6a4f290f5"}
Sep 18 19:33:48.783577 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.783399    2580 scope.go:117] "RemoveContainer" containerID="8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"
Sep 18 19:33:48.792145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.792126    2580 scope.go:117] "RemoveContainer" containerID="8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"
Sep 18 19:33:48.792419 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:33:48.792400    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec\": container with ID starting with 8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec not found: ID does not exist" containerID="8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"
Sep 18 19:33:48.792490 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.792425    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec"} err="failed to get container status \"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec\": rpc error: code = NotFound desc = could not find container \"8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec\": container with ID starting with 8eb11108fd42c390743e197434da89e7a7482fe4d932a7f9c6ae5c7f08735bec not found: ID does not exist"
Sep 18 19:33:48.806627 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.806602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:33:48.810075 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:48.810055    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s5qnl"]
Sep 18 19:33:49.294763 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:49.294733    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" path="/var/lib/kubelet/pods/4d8cea04-854b-45b0-9f17-2784e5591ae5/volumes"
Sep 18 19:33:52.484416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:52.484385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:52.587776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:52.587748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:59.015378 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:33:59.015340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:00.823883 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:00.823847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:01.058574 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.058538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:34:01.059031 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.059010    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main"
Sep 18 19:34:01.059031 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.059030    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main"
Sep 18 19:34:01.059229 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.059184    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8cea04-854b-45b0-9f17-2784e5591ae5" containerName="main"
Sep 18 19:34:01.062298 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.062278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.067811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.067790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9jv9p\""
Sep 18 19:34:01.067912 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.067854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 19:34:01.067975 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.067914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 19:34:01.068035 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.068009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 19:34:01.068174 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.068158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-6xxsg\""
Sep 18 19:34:01.078177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.078117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:34:01.104303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.104278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:01.149416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.149388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.149575 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.149455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hphj\" (UniqueName: \"kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.250658 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.250622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.250827 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.250695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hphj\" (UniqueName: \"kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.254226 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.254197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 19:34:01.261125 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.261103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 19:34:01.263740 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.263707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.271038 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.271017    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:34:01.271152 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.271051    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:34:01.271691 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.271671    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 19:34:01.280492 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.280473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hphj\" (UniqueName: \"kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.377746 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.377715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-6xxsg\""
Sep 18 19:34:01.388103 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.388078    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9jv9p\""
Sep 18 19:34:01.395416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.395396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.670834 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.670810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:34:01.672891 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:34:01.672861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b1c07f_5d67_443e_92af_527a66c3431c.slice/crio-6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b WatchSource:0}: Error finding container 6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b: Status 404 returned error can't find the container with id 6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b
Sep 18 19:34:01.846881 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.846842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" event={"ID":"68b1c07f-5d67-443e-92af-527a66c3431c","Type":"ContainerStarted","Data":"b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040"}
Sep 18 19:34:01.847285 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.846895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" event={"ID":"68b1c07f-5d67-443e-92af-527a66c3431c","Type":"ContainerStarted","Data":"6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b"}
Sep 18 19:34:01.847285 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.846952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:01.869444 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:01.869386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" podStartSLOduration=0.86936761 podStartE2EDuration="869.36761ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:01.868512324 +0000 UTC m=+1202.655792728" watchObservedRunningTime="2026-09-18 19:34:01.86936761 +0000 UTC m=+1202.656648051"
Sep 18 19:34:05.037881 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:05.037851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:05.152575 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:05.152546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:05.423883 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:05.423850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:19.485792 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:19.485758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:19.598727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:19.598688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:25.088811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:25.088770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.021230 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:26.021196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.099423 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:26.099393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:32.855486 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:32.855458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:34:35.982178 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:35.982124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:36.274142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.274056    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:34:36.278211 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.278189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.288795 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.288772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cpu-accelerator-preset-ser-afc820a4-epp-sa-dockercfg-45t2z\""
Sep 18 19:34:36.288913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.288772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 18 19:34:36.288913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.288822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 18 19:34:36.288913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.288840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 18 19:34:36.288913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.288771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-g2ltm\""
Sep 18 19:34:36.305401 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.305377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:34:36.314999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.314978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:36.459850 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.459816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.460041 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.459874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgnj\" (UniqueName: \"kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.560734 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.560703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.560908 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.560772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgnj\" (UniqueName: \"kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.563307 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.563278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.579372 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.579348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgnj\" (UniqueName: \"kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.588837 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.588818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:36.864726 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:36.864699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:34:36.866758 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:34:36.866733    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09f2481_161c_4fdf_966f_50fb6f410695.slice/crio-188c9dd902b95df71382af2afa15cf2865da01b792b3e3417369f8d13d4db1af WatchSource:0}: Error finding container 188c9dd902b95df71382af2afa15cf2865da01b792b3e3417369f8d13d4db1af: Status 404 returned error can't find the container with id 188c9dd902b95df71382af2afa15cf2865da01b792b3e3417369f8d13d4db1af
Sep 18 19:34:37.034075 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:37.034039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" event={"ID":"a09f2481-161c-4fdf-966f-50fb6f410695","Type":"ContainerStarted","Data":"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"}
Sep 18 19:34:37.034526 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:37.034095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" event={"ID":"a09f2481-161c-4fdf-966f-50fb6f410695","Type":"ContainerStarted","Data":"188c9dd902b95df71382af2afa15cf2865da01b792b3e3417369f8d13d4db1af"}
Sep 18 19:34:37.034526 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:37.034122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:34:37.056018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:34:37.055971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podStartSLOduration=1.055955812 podStartE2EDuration="1.055955812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:37.054614598 +0000 UTC m=+1237.841895003" watchObservedRunningTime="2026-09-18 19:34:37.055955812 +0000 UTC m=+1237.843236214"
Sep 18 19:35:07.772645 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:07.772616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:08.041206 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:08.041130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:35:12.437914 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:12.437876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:12.438636 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:12.438611    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:35:12.640188 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:12.640150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:12.856252 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:12.856215    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:22.856147 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:22.856108    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:24.122761 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.122338    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 18 19:35:24.122761 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:35:24.122370    2580 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main" containerID="cri-o://b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040"
Sep 18 19:35:24.122761 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.122414    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main" containerID="cri-o://b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040" gracePeriod=49
Sep 18 19:35:24.270144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.266343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl_68b1c07f-5d67-443e-92af-527a66c3431c/main/0.log"
Sep 18 19:35:24.270144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.266402    2580 generic.go:356] "Generic (PLEG): container finished" podID="68b1c07f-5d67-443e-92af-527a66c3431c" containerID="b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040" exitCode=1
Sep 18 19:35:24.270144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.266463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" event={"ID":"68b1c07f-5d67-443e-92af-527a66c3431c","Type":"ContainerDied","Data":"b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040"}
Sep 18 19:35:24.270144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.266566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl" event={"ID":"68b1c07f-5d67-443e-92af-527a66c3431c","Type":"ContainerDied","Data":"6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b"}
Sep 18 19:35:24.270144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.266616    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d79b263ccff309c6d9d53ee85d4767813f83fc5440e1749bdacee2654b69c2b"
Sep 18 19:35:24.274178 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.274156    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl_68b1c07f-5d67-443e-92af-527a66c3431c/main/0.log"
Sep 18 19:35:24.276502 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.276484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:35:24.285357 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.285334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hphj\" (UniqueName: \"kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj\") pod \"68b1c07f-5d67-443e-92af-527a66c3431c\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") "
Sep 18 19:35:24.285509 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.285373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs\") pod \"68b1c07f-5d67-443e-92af-527a66c3431c\" (UID: \"68b1c07f-5d67-443e-92af-527a66c3431c\") "
Sep 18 19:35:24.287572 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.287546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj" (OuterVolumeSpecName: "kube-api-access-7hphj") pod "68b1c07f-5d67-443e-92af-527a66c3431c" (UID: "68b1c07f-5d67-443e-92af-527a66c3431c"). InnerVolumeSpecName "kube-api-access-7hphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:35:24.287572 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.287563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68b1c07f-5d67-443e-92af-527a66c3431c" (UID: "68b1c07f-5d67-443e-92af-527a66c3431c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:35:24.386091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.386010    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1c07f-5d67-443e-92af-527a66c3431c-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:35:24.386091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:24.386036    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hphj\" (UniqueName: \"kubernetes.io/projected/68b1c07f-5d67-443e-92af-527a66c3431c-kube-api-access-7hphj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:35:25.271226 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:25.271197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"
Sep 18 19:35:25.296018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:25.295962    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:35:25.298315 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:25.298294    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bg5gl"]
Sep 18 19:35:27.294910 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:27.294880    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" path="/var/lib/kubelet/pods/68b1c07f-5d67-443e-92af-527a66c3431c/volumes"
Sep 18 19:35:53.449494 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:35:53.449452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:00.675026 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:00.674986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:00.703918 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:00.703881    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:36:00.773249 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:00.773221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:01.012914 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:01.012824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:01.107697 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:01.107664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:07.636384 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:07.636347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:08.041090 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:08.041010    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:36:09.316950 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.316909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:09.560649 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.560611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:36:09.561217 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.561198    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main"
Sep 18 19:36:09.561281 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.561220    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main"
Sep 18 19:36:09.561349 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.561338    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b1c07f-5d67-443e-92af-527a66c3431c" containerName="main"
Sep 18 19:36:09.565007 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.564981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.570995 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.570922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-hhn8n\""
Sep 18 19:36:09.571130 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.571076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 19:36:09.571130 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.571090    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 19:36:09.571265 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.571161    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-2s8sv\""
Sep 18 19:36:09.571375 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.571353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 19:36:09.579957 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.579923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:36:09.681711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.681672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.681889 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.681726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.681889 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.681779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.681889 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.681812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hbs\" (UniqueName: \"kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783085 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783300 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hbs\" (UniqueName: \"kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.783628 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.783523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.785634 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.785618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.792862 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.792835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hbs\" (UniqueName: \"kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:09.876536 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:09.876448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:10.160177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:10.160103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:36:10.162867 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:36:10.162835    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e4754a_6924_4193_b205_65ced3c281df.slice/crio-d6fb678146a907b498559172de1379e6518931b07027810f226fbdccac1230ed WatchSource:0}: Error finding container d6fb678146a907b498559172de1379e6518931b07027810f226fbdccac1230ed: Status 404 returned error can't find the container with id d6fb678146a907b498559172de1379e6518931b07027810f226fbdccac1230ed
Sep 18 19:36:10.499460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:10.499376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" event={"ID":"34e4754a-6924-4193-b205-65ced3c281df","Type":"ContainerStarted","Data":"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"}
Sep 18 19:36:10.499460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:10.499428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" event={"ID":"34e4754a-6924-4193-b205-65ced3c281df","Type":"ContainerStarted","Data":"d6fb678146a907b498559172de1379e6518931b07027810f226fbdccac1230ed"}
Sep 18 19:36:10.499460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:10.499457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:10.530314 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:10.530265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podStartSLOduration=1.5302449299999998 podStartE2EDuration="1.53024493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:10.52969234 +0000 UTC m=+1331.316972746" watchObservedRunningTime="2026-09-18 19:36:10.53024493 +0000 UTC m=+1331.317525337"
Sep 18 19:36:11.084023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:11.083992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:13.420932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:13.420845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:13.523302 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:13.523271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:15.723978 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:15.723918    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main" containerID="cri-o://45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b" gracePeriod=45
Sep 18 19:36:18.041303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:18.041257    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:36:28.040852 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:28.040811    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:36:28.041273 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:28.040922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:36:28.235094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:28.235055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:38.040985 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:38.040938    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:36:41.505159 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:41.505129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:36:43.032073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:43.032034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:43.043171 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:43.043139    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:36:43.132939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:43.132905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:45.987246 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:45.987185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:36:46.099319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.099281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs\") pod \"a09f2481-161c-4fdf-966f-50fb6f410695\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") "
Sep 18 19:36:46.099519 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.099338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mgnj\" (UniqueName: \"kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj\") pod \"a09f2481-161c-4fdf-966f-50fb6f410695\" (UID: \"a09f2481-161c-4fdf-966f-50fb6f410695\") "
Sep 18 19:36:46.101607 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.101568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a09f2481-161c-4fdf-966f-50fb6f410695" (UID: "a09f2481-161c-4fdf-966f-50fb6f410695"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:46.101726 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.101685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj" (OuterVolumeSpecName: "kube-api-access-2mgnj") pod "a09f2481-161c-4fdf-966f-50fb6f410695" (UID: "a09f2481-161c-4fdf-966f-50fb6f410695"). InnerVolumeSpecName "kube-api-access-2mgnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:46.200291 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.200253    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f2481-161c-4fdf-966f-50fb6f410695-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:36:46.200291 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.200283    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mgnj\" (UniqueName: \"kubernetes.io/projected/a09f2481-161c-4fdf-966f-50fb6f410695-kube-api-access-2mgnj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:36:46.685248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.685214    2580 generic.go:356] "Generic (PLEG): container finished" podID="a09f2481-161c-4fdf-966f-50fb6f410695" containerID="45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b" exitCode=0
Sep 18 19:36:46.685426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.685259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"
Sep 18 19:36:46.685426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.685282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" event={"ID":"a09f2481-161c-4fdf-966f-50fb6f410695","Type":"ContainerDied","Data":"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"}
Sep 18 19:36:46.685426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.685323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b" event={"ID":"a09f2481-161c-4fdf-966f-50fb6f410695","Type":"ContainerDied","Data":"188c9dd902b95df71382af2afa15cf2865da01b792b3e3417369f8d13d4db1af"}
Sep 18 19:36:46.685426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.685344    2580 scope.go:117] "RemoveContainer" containerID="45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"
Sep 18 19:36:46.694124 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.694103    2580 scope.go:117] "RemoveContainer" containerID="45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"
Sep 18 19:36:46.694398 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:36:46.694379    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b\": container with ID starting with 45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b not found: ID does not exist" containerID="45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"
Sep 18 19:36:46.694445 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.694409    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b"} err="failed to get container status \"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b\": rpc error: code = NotFound desc = could not find container \"45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b\": container with ID starting with 45e8f1a354eef0f462773a9ea68d6e4529765179dad02ef0a90e94439f8fc82b not found: ID does not exist"
Sep 18 19:36:46.712076 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.712041    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:36:46.713937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:46.713908    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schelwl2b"]
Sep 18 19:36:47.295067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:47.295038    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" path="/var/lib/kubelet/pods/a09f2481-161c-4fdf-966f-50fb6f410695/volumes"
Sep 18 19:36:51.505590 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:51.505555    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:36:52.206788 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:52.206756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:52.439340 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:52.439305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:58.062484 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:58.062423    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main" containerID="cri-o://998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239" gracePeriod=45
Sep 18 19:36:59.461980 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:36:59.461939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:01.505875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:01.505841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:37:03.756679 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:03.756635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:04.009399 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:04.009305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:06.297807 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:06.297770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:06.393977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:06.393944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:07.885134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:07.885097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:07.984930 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:07.984894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:11.506007 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:11.505971    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:37:11.506507 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:11.506092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:37:21.506279 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:21.506238    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:37:28.315502 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.315473    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:37:28.365737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.365673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs\") pod \"34e4754a-6924-4193-b205-65ced3c281df\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") "
Sep 18 19:37:28.365856 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.365740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6hbs\" (UniqueName: \"kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs\") pod \"34e4754a-6924-4193-b205-65ced3c281df\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") "
Sep 18 19:37:28.365856 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.365763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache\") pod \"34e4754a-6924-4193-b205-65ced3c281df\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") "
Sep 18 19:37:28.365856 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.365806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache\") pod \"34e4754a-6924-4193-b205-65ced3c281df\" (UID: \"34e4754a-6924-4193-b205-65ced3c281df\") "
Sep 18 19:37:28.366269 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.366042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "34e4754a-6924-4193-b205-65ced3c281df" (UID: "34e4754a-6924-4193-b205-65ced3c281df"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:37:28.366269 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.366121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "34e4754a-6924-4193-b205-65ced3c281df" (UID: "34e4754a-6924-4193-b205-65ced3c281df"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:37:28.368127 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.368105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34e4754a-6924-4193-b205-65ced3c281df" (UID: "34e4754a-6924-4193-b205-65ced3c281df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:37:28.368352 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.368335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs" (OuterVolumeSpecName: "kube-api-access-b6hbs") pod "34e4754a-6924-4193-b205-65ced3c281df" (UID: "34e4754a-6924-4193-b205-65ced3c281df"). InnerVolumeSpecName "kube-api-access-b6hbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:37:28.466641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.466613    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4754a-6924-4193-b205-65ced3c281df-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:37:28.466641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.466637    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hbs\" (UniqueName: \"kubernetes.io/projected/34e4754a-6924-4193-b205-65ced3c281df-kube-api-access-b6hbs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:37:28.466641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.466649    2580 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-tokenizer-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:37:28.466866 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.466658    2580 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4754a-6924-4193-b205-65ced3c281df-cachi2-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:37:28.901341 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.901305    2580 generic.go:356] "Generic (PLEG): container finished" podID="34e4754a-6924-4193-b205-65ced3c281df" containerID="998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239" exitCode=0
Sep 18 19:37:28.901510 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.901347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" event={"ID":"34e4754a-6924-4193-b205-65ced3c281df","Type":"ContainerDied","Data":"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"}
Sep 18 19:37:28.901510 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.901363    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"
Sep 18 19:37:28.901510 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.901421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d" event={"ID":"34e4754a-6924-4193-b205-65ced3c281df","Type":"ContainerDied","Data":"d6fb678146a907b498559172de1379e6518931b07027810f226fbdccac1230ed"}
Sep 18 19:37:28.901510 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.901444    2580 scope.go:117] "RemoveContainer" containerID="998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"
Sep 18 19:37:28.910875 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.910858    2580 scope.go:117] "RemoveContainer" containerID="998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"
Sep 18 19:37:28.911136 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:37:28.911115    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239\": container with ID starting with 998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239 not found: ID does not exist" containerID="998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"
Sep 18 19:37:28.911183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.911144    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239"} err="failed to get container status \"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239\": rpc error: code = NotFound desc = could not find container \"998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239\": container with ID starting with 998368b255dd837521beaf55d97f077cea83e8922c33ec5b23af51ae72579239 not found: ID does not exist"
Sep 18 19:37:28.933427 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.933400    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:37:28.935208 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:28.935191    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc2rx9d"]
Sep 18 19:37:29.294808 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:29.294734    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e4754a-6924-4193-b205-65ced3c281df" path="/var/lib/kubelet/pods/34e4754a-6924-4193-b205-65ced3c281df/volumes"
Sep 18 19:37:33.044314 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:33.044283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:33.166900 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:33.166862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:39.730718 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:39.730671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:40.505606 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.505559    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:37:40.506041 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506021    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main"
Sep 18 19:37:40.506134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main"
Sep 18 19:37:40.506134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506084    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main"
Sep 18 19:37:40.506134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506093    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main"
Sep 18 19:37:40.506292 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506183    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e4754a-6924-4193-b205-65ced3c281df" containerName="main"
Sep 18 19:37:40.506292 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.506201    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09f2481-161c-4fdf-966f-50fb6f410695" containerName="main"
Sep 18 19:37:40.510671 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.510651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.518772 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.518755    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 19:37:40.519359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.519342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 19:37:40.520600 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.520571    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 19:37:40.521023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.521010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-m29z2\""
Sep 18 19:37:40.533744 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.533717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:37:40.563308 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.563274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:40.573122 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.573245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.573245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.573345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.573345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2fp\" (UniqueName: \"kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.573440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.573345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2fp\" (UniqueName: \"kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674666 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.674999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.674930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.676833 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.676814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.676950 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.676933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.685275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.685249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2fp\" (UniqueName: \"kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.812965 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.812934    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:37:40.816842 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.816820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.820438 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.820233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-tc9cc\""
Sep 18 19:37:40.820438 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.820396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:40.830432 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.830408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:37:40.860321 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.860284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:40.877028 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.877001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.877196 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.877032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjsx\" (UniqueName: \"kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.977960 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.977928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.977960 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.977959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjsx\" (UniqueName: \"kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.980467 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.980443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:40.987442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:40.987413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjsx\" (UniqueName: \"kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:41.109185 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.109157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:37:41.111271 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:37:41.111241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af39d0a_4f87_4728_b8da_0fff03b7282a.slice/crio-76a0a5c16c977ef4827bacace3534dde5d13a7698b193e124c8293b1689707e8 WatchSource:0}: Error finding container 76a0a5c16c977ef4827bacace3534dde5d13a7698b193e124c8293b1689707e8: Status 404 returned error can't find the container with id 76a0a5c16c977ef4827bacace3534dde5d13a7698b193e124c8293b1689707e8
Sep 18 19:37:41.113251 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.113231    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:37:41.129895 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.129867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:41.417873 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.417847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:37:41.420106 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:37:41.420076    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9556ee_a73a_4d5f_803e_8d7795af7e5f.slice/crio-3c4239b710fa1ba43156d48bfa5d80210e99e1b363431b995f0161798a5bc172 WatchSource:0}: Error finding container 3c4239b710fa1ba43156d48bfa5d80210e99e1b363431b995f0161798a5bc172: Status 404 returned error can't find the container with id 3c4239b710fa1ba43156d48bfa5d80210e99e1b363431b995f0161798a5bc172
Sep 18 19:37:41.975376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.975336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" event={"ID":"eb9556ee-a73a-4d5f-803e-8d7795af7e5f","Type":"ContainerStarted","Data":"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"}
Sep 18 19:37:41.975930 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.975404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" event={"ID":"eb9556ee-a73a-4d5f-803e-8d7795af7e5f","Type":"ContainerStarted","Data":"3c4239b710fa1ba43156d48bfa5d80210e99e1b363431b995f0161798a5bc172"}
Sep 18 19:37:41.975930 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.975618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:37:41.977345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:41.977309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" event={"ID":"5af39d0a-4f87-4728-b8da-0fff03b7282a","Type":"ContainerStarted","Data":"76a0a5c16c977ef4827bacace3534dde5d13a7698b193e124c8293b1689707e8"}
Sep 18 19:37:42.983574 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:42.983473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" event={"ID":"5af39d0a-4f87-4728-b8da-0fff03b7282a","Type":"ContainerStarted","Data":"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"}
Sep 18 19:37:43.820755 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:43.820670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:44.160939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:44.160843    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podStartSLOduration=4.16082806 podStartE2EDuration="4.16082806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:43.787874118 +0000 UTC m=+1424.575154522" watchObservedRunningTime="2026-09-18 19:37:44.16082806 +0000 UTC m=+1424.948108464"
Sep 18 19:37:44.161293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:44.161231    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" podStartSLOduration=2.684556743 podStartE2EDuration="4.161224376s" totalImagesPullingTime="1.476667633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:44.159917274 +0000 UTC m=+1424.947197677" watchObservedRunningTime="2026-09-18 19:37:44.161224376 +0000 UTC m=+1424.948504776"
Sep 18 19:37:50.820803 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:50.820774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:50.833075 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:50.833049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:51.034409 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:51.034377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:37:52.895177 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:52.895142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:52.989415 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:52.989383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:55.751073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:37:55.751036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:05.541316 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:38:05.541275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:05.636223 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:38:05.636182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:12.984775 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:38:12.984749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:38:54.085614 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:38:54.085558    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:54.202309 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:38:54.202275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:00.967709 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:00.967677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:01.303132 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:01.303103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:39:01.303769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:01.303748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:39:03.164490 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:03.164455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.401334 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:03.401298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.825462 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:06.825406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.899931 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:06.899898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.306391 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:09.306356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.610841 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.610808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.630696 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.630665    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:39:20.640592 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.640553    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:39:20.650280 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:39:20.650247    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:39:20.650280 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.650280 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.650280 ip-10-0-138-2 kubenswrapper[2580]:  > containerID="c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411" cmd=["/bin/sleep","15"]
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.650306    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" message="" err=<
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:  >
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:39:20.650322    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerName="main" containerID="cri-o://c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"
Sep 18 19:39:20.650573 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.650367    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerName="main" containerID="cri-o://c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411" gracePeriod=60
Sep 18 19:39:20.686441 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:20.686412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:21.001916 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.001887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:39:21.142004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.141914    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.141964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.141984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2fp\" (UniqueName: \"kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp\") pod \"5af39d0a-4f87-4728-b8da-0fff03b7282a\" (UID: \"5af39d0a-4f87-4728-b8da-0fff03b7282a\") "
Sep 18 19:39:21.142287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.142287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.142473 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142361    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.142473 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.142375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home" (OuterVolumeSpecName: "home") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.144157 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.144133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm" (OuterVolumeSpecName: "dshm") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:21.144262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.144219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:21.144337 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.144253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp" (OuterVolumeSpecName: "kube-api-access-vx2fp") pod "5af39d0a-4f87-4728-b8da-0fff03b7282a" (UID: "5af39d0a-4f87-4728-b8da-0fff03b7282a"). InnerVolumeSpecName "kube-api-access-vx2fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:21.243668 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.243637    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af39d0a-4f87-4728-b8da-0fff03b7282a-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.243668 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.243664    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.243668 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.243675    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2fp\" (UniqueName: \"kubernetes.io/projected/5af39d0a-4f87-4728-b8da-0fff03b7282a-kube-api-access-vx2fp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.243888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.243686    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.243888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.243695    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af39d0a-4f87-4728-b8da-0fff03b7282a-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:39:21.472163 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.472084    2580 generic.go:356] "Generic (PLEG): container finished" podID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerID="c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411" exitCode=0
Sep 18 19:39:21.472163 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.472132    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"
Sep 18 19:39:21.472348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.472176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" event={"ID":"5af39d0a-4f87-4728-b8da-0fff03b7282a","Type":"ContainerDied","Data":"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"}
Sep 18 19:39:21.472348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.472232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5" event={"ID":"5af39d0a-4f87-4728-b8da-0fff03b7282a","Type":"ContainerDied","Data":"76a0a5c16c977ef4827bacace3534dde5d13a7698b193e124c8293b1689707e8"}
Sep 18 19:39:21.472348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.472260    2580 scope.go:117] "RemoveContainer" containerID="c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"
Sep 18 19:39:21.481345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.481325    2580 scope.go:117] "RemoveContainer" containerID="c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"
Sep 18 19:39:21.481638 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:39:21.481620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411\": container with ID starting with c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411 not found: ID does not exist" containerID="c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"
Sep 18 19:39:21.481690 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.481646    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411"} err="failed to get container status \"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411\": rpc error: code = NotFound desc = could not find container \"c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411\": container with ID starting with c859248236e20562dd0b797f09705dd30ce0e12d0d7c69725f44b9fa5e928411 not found: ID does not exist"
Sep 18 19:39:21.492184 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.492164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:39:21.496233 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:21.496212    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5lbc5"]
Sep 18 19:39:22.985860 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:22.985823    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:39:23.301502 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:23.301461    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" path="/var/lib/kubelet/pods/5af39d0a-4f87-4728-b8da-0fff03b7282a/volumes"
Sep 18 19:39:32.985825 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:32.985787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:39:35.650339 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:35.650270    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" containerID="cri-o://a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916" gracePeriod=45
Sep 18 19:39:42.766430 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:42.766395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.869566 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:42.869531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.985764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:42.985722    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:39:42.985939 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:42.985819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:39:52.985501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:39:52.985460    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:40:02.985442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:02.985390    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:40:05.899447 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:05.899424    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:40:06.022516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.022432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztjsx\" (UniqueName: \"kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx\") pod \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") "
Sep 18 19:40:06.022516 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.022493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs\") pod \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\" (UID: \"eb9556ee-a73a-4d5f-803e-8d7795af7e5f\") "
Sep 18 19:40:06.024595 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.024556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb9556ee-a73a-4d5f-803e-8d7795af7e5f" (UID: "eb9556ee-a73a-4d5f-803e-8d7795af7e5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:06.024704 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.024556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx" (OuterVolumeSpecName: "kube-api-access-ztjsx") pod "eb9556ee-a73a-4d5f-803e-8d7795af7e5f" (UID: "eb9556ee-a73a-4d5f-803e-8d7795af7e5f"). InnerVolumeSpecName "kube-api-access-ztjsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:06.123145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.123117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjsx\" (UniqueName: \"kubernetes.io/projected/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-kube-api-access-ztjsx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.123145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.123143    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9556ee-a73a-4d5f-803e-8d7795af7e5f-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.699542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.699510    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerID="a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916" exitCode=0
Sep 18 19:40:06.699729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.699550    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"
Sep 18 19:40:06.699729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.699563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" event={"ID":"eb9556ee-a73a-4d5f-803e-8d7795af7e5f","Type":"ContainerDied","Data":"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"}
Sep 18 19:40:06.699729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.699620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5" event={"ID":"eb9556ee-a73a-4d5f-803e-8d7795af7e5f","Type":"ContainerDied","Data":"3c4239b710fa1ba43156d48bfa5d80210e99e1b363431b995f0161798a5bc172"}
Sep 18 19:40:06.699729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.699642    2580 scope.go:117] "RemoveContainer" containerID="a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"
Sep 18 19:40:06.709346 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.709330    2580 scope.go:117] "RemoveContainer" containerID="a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"
Sep 18 19:40:06.709791 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:40:06.709768    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916\": container with ID starting with a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916 not found: ID does not exist" containerID="a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"
Sep 18 19:40:06.709874 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.709799    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916"} err="failed to get container status \"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916\": rpc error: code = NotFound desc = could not find container \"a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916\": container with ID starting with a8d5213e3e53420469dfc8dc380a3160b3583ca9de86fbe606284e37ed96f916 not found: ID does not exist"
Sep 18 19:40:06.724650 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.724627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:40:06.728541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:06.728519    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758q8f5"]
Sep 18 19:40:07.297468 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:07.297414    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" path="/var/lib/kubelet/pods/eb9556ee-a73a-4d5f-803e-8d7795af7e5f/volumes"
Sep 18 19:40:10.719240 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:10.719194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:10.968740 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:10.968706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:18.157601 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:18.157557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:21.375627 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.375570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:21.610394 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610363    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:40:21.610827 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerName="main"
Sep 18 19:40:21.610888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerName="main"
Sep 18 19:40:21.610888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610845    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main"
Sep 18 19:40:21.610888 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610850    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main"
Sep 18 19:40:21.610979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610922    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af39d0a-4f87-4728-b8da-0fff03b7282a" containerName="main"
Sep 18 19:40:21.610979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.610931    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9556ee-a73a-4d5f-803e-8d7795af7e5f" containerName="main"
Sep 18 19:40:21.615280 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.615264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.641190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.641118    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 19:40:21.641190 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.641184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-r66hn\""
Sep 18 19:40:21.641411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.641201    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 19:40:21.641411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.641118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-p8q8f\""
Sep 18 19:40:21.641411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.641118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 19:40:21.643460 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.643438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:40:21.682500 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.682469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:21.754136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.754100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnz7r\" (UniqueName: \"kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.754294 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.754150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.855491 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.855443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnz7r\" (UniqueName: \"kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.855729 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.855513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.858081 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.858056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.865335 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.865311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnz7r\" (UniqueName: \"kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched82djc\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:21.926262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:21.926190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:22.214988 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:22.214893    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:40:22.218224 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:40:22.218187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404210d5_4b64_49dc_9a8c_a6c2181452e4.slice/crio-b318c69f4f1985129027df36a6063c128fc96c321f26fc928704038a616c6ca9 WatchSource:0}: Error finding container b318c69f4f1985129027df36a6063c128fc96c321f26fc928704038a616c6ca9: Status 404 returned error can't find the container with id b318c69f4f1985129027df36a6063c128fc96c321f26fc928704038a616c6ca9
Sep 18 19:40:22.786633 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:22.786602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" event={"ID":"404210d5-4b64-49dc-9a8c-a6c2181452e4","Type":"ContainerStarted","Data":"53e6b00ea22577babcf778b02f081e3bfd0424b5e90eb5d23c4be9dd832ab828"}
Sep 18 19:40:22.787004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:22.786650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" event={"ID":"404210d5-4b64-49dc-9a8c-a6c2181452e4","Type":"ContainerStarted","Data":"b318c69f4f1985129027df36a6063c128fc96c321f26fc928704038a616c6ca9"}
Sep 18 19:40:22.787004 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:22.786748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:40:22.807913 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:22.807867    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podStartSLOduration=1.807853946 podStartE2EDuration="1.807853946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:22.806242468 +0000 UTC m=+1583.593522873" watchObservedRunningTime="2026-09-18 19:40:22.807853946 +0000 UTC m=+1583.595134349"
Sep 18 19:40:24.558960 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:24.558925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:24.641999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:24.641937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:32.184638 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:32.184566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:32.332993 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:32.332935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:32.520142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:32.520041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:32.893819 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:32.893783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:33.111439 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:33.111403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:36.648541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:36.648498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:39.104296 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:39.104260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:45.081918 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:45.081831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:45.158272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:45.158241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:45.895094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:45.895054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:53.793342 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:40:53.793305    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:41:01.398529 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:01.398493    2580 scope.go:117] "RemoveContainer" containerID="b8355919f0fc52bc4ea23f53cb9d5310e4b9f07280717985135c33eab0c94040"
Sep 18 19:41:15.243306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:15.243266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:17.015963 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:17.015932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:17.971416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:17.971377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:18.258341 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:18.258254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:18.588352 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:18.588311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:18.724630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:18.724591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:21.433637 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:21.433605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:21.850183 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:21.850149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:22.942692 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:22.942657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:29.550775 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:29.550746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:30.559153 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:30.559117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:35.891062 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:35.891026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:45.004454 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:45.004414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:45.483110 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:45.483071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:45.560432 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:45.560395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:49.611648 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:49.611613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:51.759155 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:51.759120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:51.863129 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:51.863087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:52.143540 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:52.143508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:52.203907 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:52.203871    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:41:52.469541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:52.469453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:52.707514 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:52.707481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:53.793391 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:53.793351    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:41:56.805891 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:41:56.805859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:02.441949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:02.441909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:02.776577 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:02.776498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:03.793381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:03.793348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:42:03.820383 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:03.820350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.771173 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:04.771141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:07.224039 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:07.224000    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" containerID="cri-o://53e6b00ea22577babcf778b02f081e3bfd0424b5e90eb5d23c4be9dd832ab828" gracePeriod=45
Sep 18 19:42:08.338716 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:08.338689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:13.793131 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:13.793052    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:42:13.793520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:13.793173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:42:14.997096 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:14.997063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:20.966142 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:20.966104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:21.198395 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:21.198362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:23.793710 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:23.793669    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:42:33.792950 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:33.792912    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:42:34.377327 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:34.377294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.456727 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.456695    2580 generic.go:356] "Generic (PLEG): container finished" podID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerID="53e6b00ea22577babcf778b02f081e3bfd0424b5e90eb5d23c4be9dd832ab828" exitCode=0
Sep 18 19:42:37.457027 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.456764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" event={"ID":"404210d5-4b64-49dc-9a8c-a6c2181452e4","Type":"ContainerDied","Data":"53e6b00ea22577babcf778b02f081e3bfd0424b5e90eb5d23c4be9dd832ab828"}
Sep 18 19:42:37.476761 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.476739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:42:37.537056 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.536958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnz7r\" (UniqueName: \"kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r\") pod \"404210d5-4b64-49dc-9a8c-a6c2181452e4\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") "
Sep 18 19:42:37.537056 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.536992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs\") pod \"404210d5-4b64-49dc-9a8c-a6c2181452e4\" (UID: \"404210d5-4b64-49dc-9a8c-a6c2181452e4\") "
Sep 18 19:42:37.539222 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.539196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "404210d5-4b64-49dc-9a8c-a6c2181452e4" (UID: "404210d5-4b64-49dc-9a8c-a6c2181452e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:37.539362 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.539275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r" (OuterVolumeSpecName: "kube-api-access-mnz7r") pod "404210d5-4b64-49dc-9a8c-a6c2181452e4" (UID: "404210d5-4b64-49dc-9a8c-a6c2181452e4"). InnerVolumeSpecName "kube-api-access-mnz7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:37.638508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.638470    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnz7r\" (UniqueName: \"kubernetes.io/projected/404210d5-4b64-49dc-9a8c-a6c2181452e4-kube-api-access-mnz7r\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:42:37.638508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:37.638499    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404210d5-4b64-49dc-9a8c-a6c2181452e4-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:42:38.462630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:38.462576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"
Sep 18 19:42:38.462630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:38.462609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc" event={"ID":"404210d5-4b64-49dc-9a8c-a6c2181452e4","Type":"ContainerDied","Data":"b318c69f4f1985129027df36a6063c128fc96c321f26fc928704038a616c6ca9"}
Sep 18 19:42:38.463107 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:38.462672    2580 scope.go:117] "RemoveContainer" containerID="53e6b00ea22577babcf778b02f081e3bfd0424b5e90eb5d23c4be9dd832ab828"
Sep 18 19:42:38.492170 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:38.492129    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:42:38.493745 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:38.493727    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched82djc"]
Sep 18 19:42:39.294174 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:39.294133    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" path="/var/lib/kubelet/pods/404210d5-4b64-49dc-9a8c-a6c2181452e4/volumes"
Sep 18 19:42:47.013332 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:47.013295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:51.676884 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:51.676838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:55.026896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:42:55.026862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:01.552231 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:01.552197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:01.655975 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:01.655939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:12.459699 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:12.459664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:15.392667 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.392634    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:15.393035 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.393002    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main"
Sep 18 19:43:15.393035 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.393014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main"
Sep 18 19:43:15.393109 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.393086    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="404210d5-4b64-49dc-9a8c-a6c2181452e4" containerName="main"
Sep 18 19:43:15.397737 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.397715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.402376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.402350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:43:15.403811 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.403787    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 19:43:15.403922 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.403789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-xlhct\""
Sep 18 19:43:15.404018 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.404002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 19:43:15.404069 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.404048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 19:43:15.404119 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.404105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 19:43:15.413061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.413034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:15.422382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.422357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:15.558701 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558701 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27mn\" (UniqueName: \"kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.558932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.558922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.661010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27mn\" (UniqueName: \"kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.661055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.661099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.661144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661868 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.661816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.661924 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.660711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.662059 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.662033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.662132 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.662114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.663308 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.663278    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:43:15.663918 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.663889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.664400 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.664380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.667157 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.667140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.670703 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.670681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-l8mhx\""
Sep 18 19:43:15.679151 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.679127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:15.682989 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.682965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27mn\" (UniqueName: \"kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn\") pod \"tokenizer-clean-path-test-kserve-5c88c58655-dfjnk\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.684134 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.684113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:43:15.708639 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.708604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:15.761920 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.761863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jc9\" (UniqueName: \"kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.762053 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.761953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.776317 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.776295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:15.862993 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.862957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jc9\" (UniqueName: \"kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.863193 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.863003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.865498 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.865472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.872900 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.872877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jc9\" (UniqueName: \"kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.989698 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.989602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:15.996611 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:15.996575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:15.999182 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:43:15.999154    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc3186e_39bd_4953_a2dd_90649bd011f7.slice/crio-14ad9cbdf7a799882d039d609822f6fc5ed456a14c586e6808ad842fe58149e8 WatchSource:0}: Error finding container 14ad9cbdf7a799882d039d609822f6fc5ed456a14c586e6808ad842fe58149e8: Status 404 returned error can't find the container with id 14ad9cbdf7a799882d039d609822f6fc5ed456a14c586e6808ad842fe58149e8
Sep 18 19:43:16.008559 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.008532    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:16.295169 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.295145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:43:16.296813 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:43:16.296788    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a7bda2_cde7_4bef_8728_0bc91656b490.slice/crio-368e6f3be99e06e2ae4234703db7923ed19802018e45855ebb5c96aad1d225cc WatchSource:0}: Error finding container 368e6f3be99e06e2ae4234703db7923ed19802018e45855ebb5c96aad1d225cc: Status 404 returned error can't find the container with id 368e6f3be99e06e2ae4234703db7923ed19802018e45855ebb5c96aad1d225cc
Sep 18 19:43:16.649624 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.649569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" event={"ID":"d2a7bda2-cde7-4bef-8728-0bc91656b490","Type":"ContainerStarted","Data":"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"}
Sep 18 19:43:16.650081 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.649647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" event={"ID":"d2a7bda2-cde7-4bef-8728-0bc91656b490","Type":"ContainerStarted","Data":"368e6f3be99e06e2ae4234703db7923ed19802018e45855ebb5c96aad1d225cc"}
Sep 18 19:43:16.650081 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.649723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:16.650836 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:16.650813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerStarted","Data":"14ad9cbdf7a799882d039d609822f6fc5ed456a14c586e6808ad842fe58149e8"}
Sep 18 19:43:20.675736 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:20.675693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerStarted","Data":"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"}
Sep 18 19:43:20.704432 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:20.704380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podStartSLOduration=5.704364267 podStartE2EDuration="5.704364267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:16.714735536 +0000 UTC m=+1757.502015941" watchObservedRunningTime="2026-09-18 19:43:20.704364267 +0000 UTC m=+1761.491644671"
Sep 18 19:43:22.689056 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:22.689023    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerID="21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990" exitCode=0
Sep 18 19:43:22.689441 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:22.689102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerDied","Data":"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"}
Sep 18 19:43:23.696025 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:23.695988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerStarted","Data":"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"}
Sep 18 19:43:23.709554 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:23.709525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:23.732984 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:23.732936    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" podStartSLOduration=2.858260615 podStartE2EDuration="8.732918281s" totalImagesPullingTime="3.656561434s" totalInitContainerRuntime="2.218096232s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:23.730096888 +0000 UTC m=+1764.517377293" watchObservedRunningTime="2026-09-18 19:43:23.732918281 +0000 UTC m=+1764.520198684"
Sep 18 19:43:25.709545 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:25.709504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:25.721991 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:25.721963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:26.721167 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:26.721136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:36.780403 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:36.780365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:47.658656 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:47.658557    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:43:52.683998 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.683965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:52.760643 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.760552    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:43:52.774690 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.774654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:52.784384 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:43:52.784344    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:43:52.784384 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.784384 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.784384 ip-10-0-138-2 kubenswrapper[2580]:  > containerID="5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877" cmd=["/bin/sleep","15"]
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.784401    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" message="" err=<
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:  >
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:43:52.784414    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="main" containerID="cri-o://5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"
Sep 18 19:43:52.784755 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.784452    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="main" containerID="cri-o://5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877" gracePeriod=60
Sep 18 19:43:52.810470 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:52.810442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:53.545202 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.545174    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:53.588700 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588700 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588701    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q27mn\" (UniqueName: \"kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.588937 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.589179 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.588958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm\") pod \"bdc3186e-39bd-4953-a2dd-90649bd011f7\" (UID: \"bdc3186e-39bd-4953-a2dd-90649bd011f7\") "
Sep 18 19:43:53.589179 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home" (OuterVolumeSpecName: "home") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.589179 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.589318 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589243    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.589318 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589260    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.589416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:53.589416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.589396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.591331 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.591302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm" (OuterVolumeSpecName: "dshm") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.591452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.591427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn" (OuterVolumeSpecName: "kube-api-access-q27mn") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "kube-api-access-q27mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:53.591515 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.591500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:53.622426 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.622376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdc3186e-39bd-4953-a2dd-90649bd011f7" (UID: "bdc3186e-39bd-4953-a2dd-90649bd011f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690669    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690701    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690711    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc3186e-39bd-4953-a2dd-90649bd011f7-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690719    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27mn\" (UniqueName: \"kubernetes.io/projected/bdc3186e-39bd-4953-a2dd-90649bd011f7-kube-api-access-q27mn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690728    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc3186e-39bd-4953-a2dd-90649bd011f7-kserve-provision-location\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.690759 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.690737    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc3186e-39bd-4953-a2dd-90649bd011f7-cabundle-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.861744 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.861708    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerID="5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877" exitCode=0
Sep 18 19:43:53.861942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.861772    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"
Sep 18 19:43:53.861942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.861787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerDied","Data":"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"}
Sep 18 19:43:53.861942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.861875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk" event={"ID":"bdc3186e-39bd-4953-a2dd-90649bd011f7","Type":"ContainerDied","Data":"14ad9cbdf7a799882d039d609822f6fc5ed456a14c586e6808ad842fe58149e8"}
Sep 18 19:43:53.861942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.861906    2580 scope.go:117] "RemoveContainer" containerID="5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"
Sep 18 19:43:53.871968 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.871951    2580 scope.go:117] "RemoveContainer" containerID="21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"
Sep 18 19:43:53.879847 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.879829    2580 scope.go:117] "RemoveContainer" containerID="5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"
Sep 18 19:43:53.880141 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:43:53.880123    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877\": container with ID starting with 5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877 not found: ID does not exist" containerID="5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"
Sep 18 19:43:53.880187 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.880151    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877"} err="failed to get container status \"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877\": rpc error: code = NotFound desc = could not find container \"5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877\": container with ID starting with 5c853649789fadd905026e38c60a0c63d778e63bc59de21a859dd95fc1cd7877 not found: ID does not exist"
Sep 18 19:43:53.880187 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.880170    2580 scope.go:117] "RemoveContainer" containerID="21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"
Sep 18 19:43:53.880410 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:43:53.880389    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990\": container with ID starting with 21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990 not found: ID does not exist" containerID="21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"
Sep 18 19:43:53.880449 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.880416    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990"} err="failed to get container status \"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990\": rpc error: code = NotFound desc = could not find container \"21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990\": container with ID starting with 21cb6ee47cc36774a2e1fd3199c86e9e42eaffe0109c2d172409315c21e24990 not found: ID does not exist"
Sep 18 19:43:53.890501 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.890474    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:53.895478 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:53.895456    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5c88c58655-dfjnk"]
Sep 18 19:43:55.295275 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:55.295238    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" path="/var/lib/kubelet/pods/bdc3186e-39bd-4953-a2dd-90649bd011f7/volumes"
Sep 18 19:43:57.658800 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:43:57.658755    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:01.332282 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:01.332247    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:44:01.335202 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:01.335179    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:44:07.658948 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:07.658904    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:07.783042 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:07.782994    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" containerID="cri-o://211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5" gracePeriod=45
Sep 18 19:44:08.130380 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:08.130342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:08.239704 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:08.239661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:17.658590 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:17.658544    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:17.659040 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:17.658696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:44:27.658834 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:27.658793    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:37.658824 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:37.658782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:38.040881 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.040859    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:44:38.067704 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.067673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5jc9\" (UniqueName: \"kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9\") pod \"d2a7bda2-cde7-4bef-8728-0bc91656b490\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") "
Sep 18 19:44:38.067863 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.067753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs\") pod \"d2a7bda2-cde7-4bef-8728-0bc91656b490\" (UID: \"d2a7bda2-cde7-4bef-8728-0bc91656b490\") "
Sep 18 19:44:38.069772 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.069747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9" (OuterVolumeSpecName: "kube-api-access-x5jc9") pod "d2a7bda2-cde7-4bef-8728-0bc91656b490" (UID: "d2a7bda2-cde7-4bef-8728-0bc91656b490"). InnerVolumeSpecName "kube-api-access-x5jc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:38.069870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.069798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2a7bda2-cde7-4bef-8728-0bc91656b490" (UID: "d2a7bda2-cde7-4bef-8728-0bc91656b490"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:38.085478 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.085449    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerID="211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5" exitCode=0
Sep 18 19:44:38.085615 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.085492    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"
Sep 18 19:44:38.085615 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.085525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" event={"ID":"d2a7bda2-cde7-4bef-8728-0bc91656b490","Type":"ContainerDied","Data":"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"}
Sep 18 19:44:38.085615 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.085598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c" event={"ID":"d2a7bda2-cde7-4bef-8728-0bc91656b490","Type":"ContainerDied","Data":"368e6f3be99e06e2ae4234703db7923ed19802018e45855ebb5c96aad1d225cc"}
Sep 18 19:44:38.085841 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.085624    2580 scope.go:117] "RemoveContainer" containerID="211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"
Sep 18 19:44:38.094405 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.094383    2580 scope.go:117] "RemoveContainer" containerID="211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"
Sep 18 19:44:38.094803 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:44:38.094673    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5\": container with ID starting with 211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5 not found: ID does not exist" containerID="211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"
Sep 18 19:44:38.094803 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.094693    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5"} err="failed to get container status \"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5\": rpc error: code = NotFound desc = could not find container \"211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5\": container with ID starting with 211d7da18a216fb4ad1c33c03f3c5c48323a3522bbeb032b1a019104954c55a5 not found: ID does not exist"
Sep 18 19:44:38.110187 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.110161    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:44:38.114041 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.114020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69hd42c"]
Sep 18 19:44:38.168952 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.168925    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5jc9\" (UniqueName: \"kubernetes.io/projected/d2a7bda2-cde7-4bef-8728-0bc91656b490-kube-api-access-x5jc9\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:44:38.168952 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:38.168951    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a7bda2-cde7-4bef-8728-0bc91656b490-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:44:39.294431 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:39.294390    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" path="/var/lib/kubelet/pods/d2a7bda2-cde7-4bef-8728-0bc91656b490/volumes"
Sep 18 19:44:42.312769 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:42.312741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:42.411938 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:42.411901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:45.333646 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.333611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:45.754508 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:44:45.754819 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="main"
Sep 18 19:44:45.754870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754821    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="main"
Sep 18 19:44:45.754870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main"
Sep 18 19:44:45.754870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754840    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main"
Sep 18 19:44:45.754870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="storage-initializer"
Sep 18 19:44:45.754870 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="storage-initializer"
Sep 18 19:44:45.755024 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754919    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a7bda2-cde7-4bef-8728-0bc91656b490" containerName="main"
Sep 18 19:44:45.755024 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.754931    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc3186e-39bd-4953-a2dd-90649bd011f7" containerName="main"
Sep 18 19:44:45.759305 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.759277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.785092 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.785064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:45.799062 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.799018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 19:44:45.799203 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.799091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-7zv9v\""
Sep 18 19:44:45.799203 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.799136    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 19:44:45.799324 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.799289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-sxvdz\""
Sep 18 19:44:45.800754 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.800737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 19:44:45.817835 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.817809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:44:45.836315 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.836274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djv4j\" (UniqueName: \"kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.836523 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.836341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.913540 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.913469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:45.937803 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.937768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.937959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.937911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djv4j\" (UniqueName: \"kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.940686 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.940664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:45.951886 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:45.951857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djv4j\" (UniqueName: \"kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:46.070085 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:46.070050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:46.350117 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:46.350042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:44:47.136171 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:47.136137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" event={"ID":"edef6a09-60bf-4b8c-8ac5-f3df716a24fc","Type":"ContainerStarted","Data":"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"}
Sep 18 19:44:47.136362 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:47.136231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" event={"ID":"edef6a09-60bf-4b8c-8ac5-f3df716a24fc","Type":"ContainerStarted","Data":"239e26bc73c70b8b26b0fc48c653f781fa11724f60fb25f51a252f9bc7c826ae"}
Sep 18 19:44:47.136362 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:47.136259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:44:47.169778 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:47.169728    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podStartSLOduration=2.169714158 podStartE2EDuration="2.169714158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:47.169133438 +0000 UTC m=+1847.956413845" watchObservedRunningTime="2026-09-18 19:44:47.169714158 +0000 UTC m=+1847.956994562"
Sep 18 19:44:50.240629 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:50.240596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:56.374461 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:56.374392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:56.449934 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:56.449899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:58.075373 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:58.075324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:58.179360 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:44:58.179331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:05.373158 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:05.373121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:08.741234 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:08.741194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:11.304659 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:11.304629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:11.400503 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:11.400463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:15.837411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:15.837316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:16.069423 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.069390    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:45:16.073894 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.073865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.078992 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.078962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 19:45:16.078992 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.078982    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-wzkdh\""
Sep 18 19:45:16.079206 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.079008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 19:45:16.079206 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.079012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 19:45:16.079206 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.079119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-nlb9d\""
Sep 18 19:45:16.086392 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.086365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:45:16.094904 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.094791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:16.219102 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.219063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.219102 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.219104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8p66\" (UniqueName: \"kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.320067 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.320027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.320247 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.320079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8p66\" (UniqueName: \"kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.322673 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.322653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.330699 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.330675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8p66\" (UniqueName: \"kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.385616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.385503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:16.665526 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:16.665493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:45:16.667603 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:45:16.667553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6122e188_1830_497f_a1d1_53eebee6e2fe.slice/crio-d7191ca5c97b7450169e67e96c82242f8f4ef9aeed89da6a8f788b9b9fbf151f WatchSource:0}: Error finding container d7191ca5c97b7450169e67e96c82242f8f4ef9aeed89da6a8f788b9b9fbf151f: Status 404 returned error can't find the container with id d7191ca5c97b7450169e67e96c82242f8f4ef9aeed89da6a8f788b9b9fbf151f
Sep 18 19:45:17.284551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:17.284516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" event={"ID":"6122e188-1830-497f-a1d1-53eebee6e2fe","Type":"ContainerStarted","Data":"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"}
Sep 18 19:45:17.284959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:17.284565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" event={"ID":"6122e188-1830-497f-a1d1-53eebee6e2fe","Type":"ContainerStarted","Data":"d7191ca5c97b7450169e67e96c82242f8f4ef9aeed89da6a8f788b9b9fbf151f"}
Sep 18 19:45:17.284959 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:17.284631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:17.314570 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:17.314515    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podStartSLOduration=1.314497811 podStartE2EDuration="1.314497811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:17.314110021 +0000 UTC m=+1878.101390426" watchObservedRunningTime="2026-09-18 19:45:17.314497811 +0000 UTC m=+1878.101778214"
Sep 18 19:45:18.143841 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:18.143807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:45:39.614156 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:39.614119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.546007 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:44.545970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.565866 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:44.565828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:45:44.627336 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:44.627304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:48.143594 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:48.143551    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:45:48.292991 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:48.292963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:45:58.142967 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:58.142923    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:45:59.587065 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:45:59.587022    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" containerID="cri-o://fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3" gracePeriod=45
Sep 18 19:46:08.142972 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:08.142932    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:46:08.143450 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:08.143033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:46:18.143398 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:18.143358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:46:28.143529 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:28.143493    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:46:29.842026 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.842001    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:46:29.891776 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.891687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs\") pod \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") "
Sep 18 19:46:29.891929 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.891820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djv4j\" (UniqueName: \"kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j\") pod \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\" (UID: \"edef6a09-60bf-4b8c-8ac5-f3df716a24fc\") "
Sep 18 19:46:29.894002 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.893968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edef6a09-60bf-4b8c-8ac5-f3df716a24fc" (UID: "edef6a09-60bf-4b8c-8ac5-f3df716a24fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:46:29.894117 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.894036    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j" (OuterVolumeSpecName: "kube-api-access-djv4j") pod "edef6a09-60bf-4b8c-8ac5-f3df716a24fc" (UID: "edef6a09-60bf-4b8c-8ac5-f3df716a24fc"). InnerVolumeSpecName "kube-api-access-djv4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:29.993485 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.993447    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:46:29.993485 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:29.993477    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djv4j\" (UniqueName: \"kubernetes.io/projected/edef6a09-60bf-4b8c-8ac5-f3df716a24fc-kube-api-access-djv4j\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:46:30.673621 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.673561    2580 generic.go:356] "Generic (PLEG): container finished" podID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerID="fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3" exitCode=0
Sep 18 19:46:30.673799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.673640    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"
Sep 18 19:46:30.673799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.673651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" event={"ID":"edef6a09-60bf-4b8c-8ac5-f3df716a24fc","Type":"ContainerDied","Data":"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"}
Sep 18 19:46:30.673799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.673734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7" event={"ID":"edef6a09-60bf-4b8c-8ac5-f3df716a24fc","Type":"ContainerDied","Data":"239e26bc73c70b8b26b0fc48c653f781fa11724f60fb25f51a252f9bc7c826ae"}
Sep 18 19:46:30.673799 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.673758    2580 scope.go:117] "RemoveContainer" containerID="fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"
Sep 18 19:46:30.683914 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.683896    2580 scope.go:117] "RemoveContainer" containerID="fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"
Sep 18 19:46:30.684218 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:46:30.684199    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3\": container with ID starting with fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3 not found: ID does not exist" containerID="fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"
Sep 18 19:46:30.684258 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.684226    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3"} err="failed to get container status \"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3\": rpc error: code = NotFound desc = could not find container \"fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3\": container with ID starting with fa8a9eef0eddbcc9b318eb6482321f92955bb15d416808cd8151e6bc681febe3 not found: ID does not exist"
Sep 18 19:46:30.722274 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.722244    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:46:30.737119 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:30.737088    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785x4gd7"]
Sep 18 19:46:31.297152 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:31.297123    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" path="/var/lib/kubelet/pods/edef6a09-60bf-4b8c-8ac5-f3df716a24fc/volumes"
Sep 18 19:46:33.307707 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:33.307652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:34.777916 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:34.777881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:40.139323 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:40.139283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:44.782382 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:44.782292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:45.034663 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.034565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:46:45.035106 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.035088    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main"
Sep 18 19:46:45.035159 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.035109    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main"
Sep 18 19:46:45.035198 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.035171    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="edef6a09-60bf-4b8c-8ac5-f3df716a24fc" containerName="main"
Sep 18 19:46:45.039775 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.039753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.045499 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.045478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 19:46:45.045650 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.045564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fbm2d\""
Sep 18 19:46:45.045733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.045601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-wgzrx\""
Sep 18 19:46:45.045733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.045661    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 19:46:45.045733 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.045480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 19:46:45.048080 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.048010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:45.052301 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.052277    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:46:45.123711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.123678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:45.133385 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.133356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qj5\" (UniqueName: \"kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.133544 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.133446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.234381 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.234347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qj5\" (UniqueName: \"kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.234551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.234420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.236973 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.236954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.244690 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.244664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qj5\" (UniqueName: \"kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.352374 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.352344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.627026 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.626999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:46:45.629104 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:46:45.629078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2029e688_cf61_4a72_b153_743e6a3ddc1f.slice/crio-1578c64e2c46677da3cb228a0eb9dbf17b5e2325de3a5a8f0fa68b9cd9eb9983 WatchSource:0}: Error finding container 1578c64e2c46677da3cb228a0eb9dbf17b5e2325de3a5a8f0fa68b9cd9eb9983: Status 404 returned error can't find the container with id 1578c64e2c46677da3cb228a0eb9dbf17b5e2325de3a5a8f0fa68b9cd9eb9983
Sep 18 19:46:45.750856 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.750822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" event={"ID":"2029e688-cf61-4a72-b153-743e6a3ddc1f","Type":"ContainerStarted","Data":"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"}
Sep 18 19:46:45.751038 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.750873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" event={"ID":"2029e688-cf61-4a72-b153-743e6a3ddc1f","Type":"ContainerStarted","Data":"1578c64e2c46677da3cb228a0eb9dbf17b5e2325de3a5a8f0fa68b9cd9eb9983"}
Sep 18 19:46:45.751038 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.750909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:46:45.773982 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:45.773934    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podStartSLOduration=0.77391755 podStartE2EDuration="773.91755ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:45.772500391 +0000 UTC m=+1966.559780795" watchObservedRunningTime="2026-09-18 19:46:45.77391755 +0000 UTC m=+1966.561197933"
Sep 18 19:46:46.920819 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:46.920788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:47.032041 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:46:47.031996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:12.392149 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:12.392101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:16.757540 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:16.757507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:47:22.829641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:22.829602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:22.854978 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:22.854938    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:47:22.914985 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:22.914953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:26.757732 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:26.757698    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:47:33.855938 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:33.855901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:36.758232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:36.758190    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:47:37.875187 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:37.875143    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" containerID="cri-o://0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b" gracePeriod=45
Sep 18 19:47:38.372449 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:38.372418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:46.757994 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:46.757952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:47:46.758465 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:46.758125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:47:49.752805 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:49.751033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:49.770546 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:49.770517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:47:49.847896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:49.847863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:56.757918 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:56.757875    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:47:58.292298 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:47:58.292256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:48:04.790411 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:04.790375    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main" containerID="cri-o://05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7" gracePeriod=45
Sep 18 19:48:06.757253 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:06.757212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:48:08.125263 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.125238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:48:08.176837 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.176737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs\") pod \"2029e688-cf61-4a72-b153-743e6a3ddc1f\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") "
Sep 18 19:48:08.176837 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.176796    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2qj5\" (UniqueName: \"kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5\") pod \"2029e688-cf61-4a72-b153-743e6a3ddc1f\" (UID: \"2029e688-cf61-4a72-b153-743e6a3ddc1f\") "
Sep 18 19:48:08.179118 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.179090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2029e688-cf61-4a72-b153-743e6a3ddc1f" (UID: "2029e688-cf61-4a72-b153-743e6a3ddc1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:08.179118 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.179106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5" (OuterVolumeSpecName: "kube-api-access-k2qj5") pod "2029e688-cf61-4a72-b153-743e6a3ddc1f" (UID: "2029e688-cf61-4a72-b153-743e6a3ddc1f"). InnerVolumeSpecName "kube-api-access-k2qj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:08.181462 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.181430    2580 generic.go:356] "Generic (PLEG): container finished" podID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerID="0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b" exitCode=0
Sep 18 19:48:08.181592 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.181482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"
Sep 18 19:48:08.181592 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.181514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" event={"ID":"2029e688-cf61-4a72-b153-743e6a3ddc1f","Type":"ContainerDied","Data":"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"}
Sep 18 19:48:08.181592 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.181574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp" event={"ID":"2029e688-cf61-4a72-b153-743e6a3ddc1f","Type":"ContainerDied","Data":"1578c64e2c46677da3cb228a0eb9dbf17b5e2325de3a5a8f0fa68b9cd9eb9983"}
Sep 18 19:48:08.181766 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.181620    2580 scope.go:117] "RemoveContainer" containerID="0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"
Sep 18 19:48:08.198076 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.198057    2580 scope.go:117] "RemoveContainer" containerID="0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"
Sep 18 19:48:08.198333 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:48:08.198313    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b\": container with ID starting with 0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b not found: ID does not exist" containerID="0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"
Sep 18 19:48:08.198384 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.198342    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b"} err="failed to get container status \"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b\": rpc error: code = NotFound desc = could not find container \"0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b\": container with ID starting with 0edafd6383ca21e2144bc714a4624f63b683c4bb304f0e714d1c96a81d9c030b not found: ID does not exist"
Sep 18 19:48:08.211538 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.211507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:48:08.217512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.217488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsxrxp"]
Sep 18 19:48:08.277569 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.277539    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2029e688-cf61-4a72-b153-743e6a3ddc1f-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.277569 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.277565    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qj5\" (UniqueName: \"kubernetes.io/projected/2029e688-cf61-4a72-b153-743e6a3ddc1f-kube-api-access-k2qj5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.292922 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:08.292887    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:48:09.294596 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:09.294553    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" path="/var/lib/kubelet/pods/2029e688-cf61-4a72-b153-743e6a3ddc1f/volumes"
Sep 18 19:48:12.883376 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:12.883346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:13.100815 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:13.100782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:18.292440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:18.292351    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:48:18.292896 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:18.292465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:48:19.949215 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:19.949128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:26.165083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:26.165043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:26.419114 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:26.419013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:26.783348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:26.783255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:26.889630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:26.889569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:28.292778 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:28.292743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:48:31.335233 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:31.335198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:35.045290 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.045262    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:48:35.218141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.218053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs\") pod \"6122e188-1830-497f-a1d1-53eebee6e2fe\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") "
Sep 18 19:48:35.218141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.218118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8p66\" (UniqueName: \"kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66\") pod \"6122e188-1830-497f-a1d1-53eebee6e2fe\" (UID: \"6122e188-1830-497f-a1d1-53eebee6e2fe\") "
Sep 18 19:48:35.220293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.220268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66" (OuterVolumeSpecName: "kube-api-access-p8p66") pod "6122e188-1830-497f-a1d1-53eebee6e2fe" (UID: "6122e188-1830-497f-a1d1-53eebee6e2fe"). InnerVolumeSpecName "kube-api-access-p8p66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:35.220293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.220284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6122e188-1830-497f-a1d1-53eebee6e2fe" (UID: "6122e188-1830-497f-a1d1-53eebee6e2fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:35.318966 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.318937    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6122e188-1830-497f-a1d1-53eebee6e2fe-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:48:35.318966 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.318969    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8p66\" (UniqueName: \"kubernetes.io/projected/6122e188-1830-497f-a1d1-53eebee6e2fe-kube-api-access-p8p66\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:48:35.322934 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.322908    2580 generic.go:356] "Generic (PLEG): container finished" podID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerID="05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7" exitCode=0
Sep 18 19:48:35.323061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.322942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" event={"ID":"6122e188-1830-497f-a1d1-53eebee6e2fe","Type":"ContainerDied","Data":"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"}
Sep 18 19:48:35.323061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.322950    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"
Sep 18 19:48:35.323061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.322983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d" event={"ID":"6122e188-1830-497f-a1d1-53eebee6e2fe","Type":"ContainerDied","Data":"d7191ca5c97b7450169e67e96c82242f8f4ef9aeed89da6a8f788b9b9fbf151f"}
Sep 18 19:48:35.323061 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.323012    2580 scope.go:117] "RemoveContainer" containerID="05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"
Sep 18 19:48:35.331748 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.331726    2580 scope.go:117] "RemoveContainer" containerID="05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"
Sep 18 19:48:35.332027 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:48:35.331999    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7\": container with ID starting with 05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7 not found: ID does not exist" containerID="05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"
Sep 18 19:48:35.332084 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.332028    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7"} err="failed to get container status \"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7\": rpc error: code = NotFound desc = could not find container \"05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7\": container with ID starting with 05b09fd8c9a080930b44b6acd5f2548945e4148c26c938a991f21b2b6933bfb7 not found: ID does not exist"
Sep 18 19:48:35.345764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.345743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:48:35.350118 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:35.350098    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnq58d"]
Sep 18 19:48:37.295179 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:37.295147    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" path="/var/lib/kubelet/pods/6122e188-1830-497f-a1d1-53eebee6e2fe/volumes"
Sep 18 19:48:39.443848 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:39.443814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:39.699193 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:39.699107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:45.940630 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:45.940600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:47.440217 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.440185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:47.687656 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.687621    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:48:47.688023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688011    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main"
Sep 18 19:48:47.688068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main"
Sep 18 19:48:47.688068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main"
Sep 18 19:48:47.688068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688039    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main"
Sep 18 19:48:47.688161 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688091    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2029e688-cf61-4a72-b153-743e6a3ddc1f" containerName="main"
Sep 18 19:48:47.688161 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.688102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6122e188-1830-497f-a1d1-53eebee6e2fe" containerName="main"
Sep 18 19:48:47.692689 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.692632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.698472 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.698448    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 19:48:47.700285 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.700267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-gmw5j\""
Sep 18 19:48:47.700404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.700317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 19:48:47.700497 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.700484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-lh7td\""
Sep 18 19:48:47.700565 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.700543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 19:48:47.706265 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.706244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:48:47.724001 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.723965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d97\" (UniqueName: \"kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.724144 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.724012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.734270 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.734238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:47.825166 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.825136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d97\" (UniqueName: \"kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.825166 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.825170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.827898 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.827874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:47.836094 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:47.836065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d97\" (UniqueName: \"kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:48.003014 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.002926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:48.279677 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.279628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:48:48.282119 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:48:48.282091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbc285e_f60d_403e_935e_fbfa53a635e4.slice/crio-28adc503d8f4e39233e7b033c0ae764f232dbbf698d347eab6a0040a1e813554 WatchSource:0}: Error finding container 28adc503d8f4e39233e7b033c0ae764f232dbbf698d347eab6a0040a1e813554: Status 404 returned error can't find the container with id 28adc503d8f4e39233e7b033c0ae764f232dbbf698d347eab6a0040a1e813554
Sep 18 19:48:48.392241 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.392210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" event={"ID":"6bbc285e-f60d-403e-935e-fbfa53a635e4","Type":"ContainerStarted","Data":"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"}
Sep 18 19:48:48.392471 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.392267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" event={"ID":"6bbc285e-f60d-403e-935e-fbfa53a635e4","Type":"ContainerStarted","Data":"28adc503d8f4e39233e7b033c0ae764f232dbbf698d347eab6a0040a1e813554"}
Sep 18 19:48:48.392471 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.392322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:48:48.415876 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:48.415828    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podStartSLOduration=1.415812351 podStartE2EDuration="1.415812351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:48.413744177 +0000 UTC m=+2089.201024597" watchObservedRunningTime="2026-09-18 19:48:48.415812351 +0000 UTC m=+2089.203092755"
Sep 18 19:48:52.694421 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:52.694376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:52.805484 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:52.805437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:57.794119 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:48:57.794082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:00.633404 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:00.633360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:00.743922 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:00.743879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:01.362045 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:01.362016    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:49:01.366213 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:01.366193    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:49:02.358443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:02.358409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:02.424997 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:02.424962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:19.404616 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:19.404567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:49:50.388148 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:50.388070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:50.483356 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:50.483320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:56.821068 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:56.821031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:57.649385 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:57.649342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:57.932199 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:49:57.932112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:01.979882 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:01.979837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:02.081324 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:02.081290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:03.497487 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:03.497455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:03.738979 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:03.738944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:03.821869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:03.821830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:11.492308 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:11.492270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:43.546279 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:43.546241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:43.863781 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:50:43.863690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:02.508920 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:02.508884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:10.349048 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:10.349012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:10.375640 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:10.375610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:51:10.582196 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:10.582162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:17.439723 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:17.439627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:17.533096 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:17.533065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:19.398781 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:19.398743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:25.394912 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:25.394873    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main" containerID="cri-o://1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736" gracePeriod=45
Sep 18 19:51:29.398571 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:29.398532    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:39.398512 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:39.398474    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:39.398977 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:39.398619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:51:49.398808 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:49.398748    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:55.662867 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.662807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:51:55.854078 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.854039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5d97\" (UniqueName: \"kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97\") pod \"6bbc285e-f60d-403e-935e-fbfa53a635e4\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") "
Sep 18 19:51:55.854248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.854128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs\") pod \"6bbc285e-f60d-403e-935e-fbfa53a635e4\" (UID: \"6bbc285e-f60d-403e-935e-fbfa53a635e4\") "
Sep 18 19:51:55.856255 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.856223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97" (OuterVolumeSpecName: "kube-api-access-v5d97") pod "6bbc285e-f60d-403e-935e-fbfa53a635e4" (UID: "6bbc285e-f60d-403e-935e-fbfa53a635e4"). InnerVolumeSpecName "kube-api-access-v5d97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:55.856368 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.856255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bbc285e-f60d-403e-935e-fbfa53a635e4" (UID: "6bbc285e-f60d-403e-935e-fbfa53a635e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:55.955471 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.955385    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5d97\" (UniqueName: \"kubernetes.io/projected/6bbc285e-f60d-403e-935e-fbfa53a635e4-kube-api-access-v5d97\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:51:55.955471 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:55.955415    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bbc285e-f60d-403e-935e-fbfa53a635e4-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.315731 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.315694    2580 generic.go:356] "Generic (PLEG): container finished" podID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerID="1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736" exitCode=0
Sep 18 19:51:56.315903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.315746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"
Sep 18 19:51:56.315903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.315787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" event={"ID":"6bbc285e-f60d-403e-935e-fbfa53a635e4","Type":"ContainerDied","Data":"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"}
Sep 18 19:51:56.315903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.315839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg" event={"ID":"6bbc285e-f60d-403e-935e-fbfa53a635e4","Type":"ContainerDied","Data":"28adc503d8f4e39233e7b033c0ae764f232dbbf698d347eab6a0040a1e813554"}
Sep 18 19:51:56.315903 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.315863    2580 scope.go:117] "RemoveContainer" containerID="1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"
Sep 18 19:51:56.324791 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.324772    2580 scope.go:117] "RemoveContainer" containerID="1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"
Sep 18 19:51:56.325057 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:51:56.325038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736\": container with ID starting with 1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736 not found: ID does not exist" containerID="1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"
Sep 18 19:51:56.325103 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.325065    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736"} err="failed to get container status \"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736\": rpc error: code = NotFound desc = could not find container \"1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736\": container with ID starting with 1501b8047f479c94d931c31db1ee7b88fcb93e83d1335c61beca6615a9528736 not found: ID does not exist"
Sep 18 19:51:56.343293 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.343255    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:51:56.345240 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:56.345216    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hjgsg"]
Sep 18 19:51:57.294505 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:51:57.294467    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" path="/var/lib/kubelet/pods/6bbc285e-f60d-403e-935e-fbfa53a635e4/volumes"
Sep 18 19:52:00.332145 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:00.332104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.457742 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:01.457699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:04.279871 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:04.279835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:06.029452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:06.029411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:08.078363 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:08.078326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:10.749948 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:10.749913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:10.852768 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:10.852731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:10.929062 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:10.929025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:14.648066 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:14.648025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:14.727221 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:14.727183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:29.559731 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:52:29.559699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:01.356220 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:01.356180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:01.532453 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:01.532418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:08.467942 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:08.467909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:09.494312 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.494284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:09.770846 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.770761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:53:09.771248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.771229    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main"
Sep 18 19:53:09.771338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.771251    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main"
Sep 18 19:53:09.771393 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.771370    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbc285e-f60d-403e-935e-fbfa53a635e4" containerName="main"
Sep 18 19:53:09.774348 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.774327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.778172 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.778152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 19:53:09.779782 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.779762    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-8p5xs\""
Sep 18 19:53:09.779879 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.779824    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ltr9b\""
Sep 18 19:53:09.780319 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.780300    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 19:53:09.781272 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.781253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 19:53:09.799064 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.799043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:53:09.837212 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.837182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:09.875556 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.875528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmnp\" (UniqueName: \"kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.875749 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.875676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.977045 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.977011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmnp\" (UniqueName: \"kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.977248 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.977083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.979545 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.979518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:09.990684 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:09.990656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmnp\" (UniqueName: \"kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:10.085359 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.085324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:10.374712 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.374684    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:53:10.375934 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:53:10.375912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9456f745_4bf1_4785_953e_a83c67c49253.slice/crio-c9f6824657ab086dfc5cf0f7b2f7a90d36fd0145f17846c62520a14901c216ba WatchSource:0}: Error finding container c9f6824657ab086dfc5cf0f7b2f7a90d36fd0145f17846c62520a14901c216ba: Status 404 returned error can't find the container with id c9f6824657ab086dfc5cf0f7b2f7a90d36fd0145f17846c62520a14901c216ba
Sep 18 19:53:10.677262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.677171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" event={"ID":"9456f745-4bf1-4785-953e-a83c67c49253","Type":"ContainerStarted","Data":"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"}
Sep 18 19:53:10.677262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.677227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" event={"ID":"9456f745-4bf1-4785-953e-a83c67c49253","Type":"ContainerStarted","Data":"c9f6824657ab086dfc5cf0f7b2f7a90d36fd0145f17846c62520a14901c216ba"}
Sep 18 19:53:10.677791 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.677308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:10.707069 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:10.707017    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podStartSLOduration=1.707002093 podStartE2EDuration="1.707002093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:10.706378334 +0000 UTC m=+2351.493658738" watchObservedRunningTime="2026-09-18 19:53:10.707002093 +0000 UTC m=+2351.494282496"
Sep 18 19:53:14.919869 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:14.919832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:15.012076 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:15.012039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:20.728005 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:20.727965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:21.891452 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:21.891417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:23.200976 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:23.200942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:23.332711 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:23.332665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:26.931038 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:26.931008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:27.213207 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:27.213124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:28.341253 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:28.341221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:36.313176 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:36.313140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:41.684710 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:41.684678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:53:49.928823 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:49.928765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:50.934944 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:50.934916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:56.953054 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:53:56.953009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:01.391861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:01.391828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:54:01.397638 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:01.397616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:54:02.150360 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.150325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:02.364101 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.364071    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:54:02.367637 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.367603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.373771 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.373743    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 19:54:02.375428 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.375407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-nks5k\""
Sep 18 19:54:02.380554 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.380531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:54:02.415443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.415376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:02.419641 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.419612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.419753 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.419725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frm8\" (UniqueName: \"kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.520239 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.520203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frm8\" (UniqueName: \"kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.520416 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.520262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.522981 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.522953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.531115 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.531092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frm8\" (UniqueName: \"kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.678083 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.677993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:02.968820 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:02.968785    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:54:02.969496 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:54:02.969470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d80d9f7_d7aa_42a1_9056_f3fc7d088fb3.slice/crio-fb55ac35f0ad7c18dc3997d08eb3986e9c3ec72eaa002794f9b7a81dceeb96b2 WatchSource:0}: Error finding container fb55ac35f0ad7c18dc3997d08eb3986e9c3ec72eaa002794f9b7a81dceeb96b2: Status 404 returned error can't find the container with id fb55ac35f0ad7c18dc3997d08eb3986e9c3ec72eaa002794f9b7a81dceeb96b2
Sep 18 19:54:03.946146 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:03.946109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" event={"ID":"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3","Type":"ContainerStarted","Data":"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"}
Sep 18 19:54:03.946660 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:03.946167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" event={"ID":"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3","Type":"ContainerStarted","Data":"fb55ac35f0ad7c18dc3997d08eb3986e9c3ec72eaa002794f9b7a81dceeb96b2"}
Sep 18 19:54:03.946660 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:03.946235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:03.975020 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:03.974962    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podStartSLOduration=1.97494993 podStartE2EDuration="1.97494993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:03.97333861 +0000 UTC m=+2404.760618993" watchObservedRunningTime="2026-09-18 19:54:03.97494993 +0000 UTC m=+2404.762230334"
Sep 18 19:54:06.173125 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:06.173080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:14.432235 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:14.432154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:15.027336 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:15.027298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:15.130325 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:15.130291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:34.954802 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:34.954766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:54:36.468002 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:36.467973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.481092 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:36.481064    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:54:36.558609 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:36.558542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:44.955091 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:44.955057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:54:51.500356 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:51.500297    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main" containerID="cri-o://9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af" gracePeriod=45
Sep 18 19:54:54.955100 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:54:54.955061    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:04.954713 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:04.954669    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:04.955101 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:04.954780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:55:14.955447 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:14.955411    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:21.747764 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.747736    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:55:21.775292 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.775215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6frm8\" (UniqueName: \"kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8\") pod \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") "
Sep 18 19:55:21.775438 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.775327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs\") pod \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\" (UID: \"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3\") "
Sep 18 19:55:21.777542 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.777517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" (UID: "0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:21.777707 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.777547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8" (OuterVolumeSpecName: "kube-api-access-6frm8") pod "0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" (UID: "0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3"). InnerVolumeSpecName "kube-api-access-6frm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:21.876812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.876784    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:55:21.876812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:21.876809    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6frm8\" (UniqueName: \"kubernetes.io/projected/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3-kube-api-access-6frm8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:55:22.361078 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.361046    2580 generic.go:356] "Generic (PLEG): container finished" podID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerID="9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af" exitCode=0
Sep 18 19:55:22.361261 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.361092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"
Sep 18 19:55:22.361261 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.361131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" event={"ID":"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3","Type":"ContainerDied","Data":"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"}
Sep 18 19:55:22.361261 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.361185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45" event={"ID":"0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3","Type":"ContainerDied","Data":"fb55ac35f0ad7c18dc3997d08eb3986e9c3ec72eaa002794f9b7a81dceeb96b2"}
Sep 18 19:55:22.361261 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.361206    2580 scope.go:117] "RemoveContainer" containerID="9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"
Sep 18 19:55:22.371264 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.371244    2580 scope.go:117] "RemoveContainer" containerID="9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"
Sep 18 19:55:22.371540 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:55:22.371520    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af\": container with ID starting with 9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af not found: ID does not exist" containerID="9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"
Sep 18 19:55:22.371676 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.371549    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af"} err="failed to get container status \"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af\": rpc error: code = NotFound desc = could not find container \"9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af\": container with ID starting with 9f563faaeda11dcfa7cca6c326f89d7f12e20c474903ece98c515ede3626c4af not found: ID does not exist"
Sep 18 19:55:22.388306 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.388278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:55:22.391786 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:22.391754    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche4rk45"]
Sep 18 19:55:23.296596 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:23.296557    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" path="/var/lib/kubelet/pods/0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3/volumes"
Sep 18 19:55:26.805657 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:26.805481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:26.834810 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:26.834772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:55:26.888369 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:26.888339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:31.684226 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:31.684182    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:33.020744 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:33.020711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:33.175595 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:33.175553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.684765 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:41.684720    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:41.854087 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:41.854040    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" containerID="cri-o://e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba" gracePeriod=45
Sep 18 19:55:51.684159 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:51.684081    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:51.684535 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:55:51.684204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:56:01.683920 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:01.683882    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:11.684659 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:11.684617    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:12.103173 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.103149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:56:12.215136 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.215100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmnp\" (UniqueName: \"kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp\") pod \"9456f745-4bf1-4785-953e-a83c67c49253\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") "
Sep 18 19:56:12.215317 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.215205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs\") pod \"9456f745-4bf1-4785-953e-a83c67c49253\" (UID: \"9456f745-4bf1-4785-953e-a83c67c49253\") "
Sep 18 19:56:12.217260 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.217230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp" (OuterVolumeSpecName: "kube-api-access-lqmnp") pod "9456f745-4bf1-4785-953e-a83c67c49253" (UID: "9456f745-4bf1-4785-953e-a83c67c49253"). InnerVolumeSpecName "kube-api-access-lqmnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:12.217378 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.217234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9456f745-4bf1-4785-953e-a83c67c49253" (UID: "9456f745-4bf1-4785-953e-a83c67c49253"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:12.316708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.316668    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9456f745-4bf1-4785-953e-a83c67c49253-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.316708 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.316702    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmnp\" (UniqueName: \"kubernetes.io/projected/9456f745-4bf1-4785-953e-a83c67c49253-kube-api-access-lqmnp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.610072 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.610039    2580 generic.go:356] "Generic (PLEG): container finished" podID="9456f745-4bf1-4785-953e-a83c67c49253" containerID="e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba" exitCode=0
Sep 18 19:56:12.610259 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.610078    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"
Sep 18 19:56:12.610259 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.610124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" event={"ID":"9456f745-4bf1-4785-953e-a83c67c49253","Type":"ContainerDied","Data":"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"}
Sep 18 19:56:12.610259 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.610171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf" event={"ID":"9456f745-4bf1-4785-953e-a83c67c49253","Type":"ContainerDied","Data":"c9f6824657ab086dfc5cf0f7b2f7a90d36fd0145f17846c62520a14901c216ba"}
Sep 18 19:56:12.610259 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.610195    2580 scope.go:117] "RemoveContainer" containerID="e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"
Sep 18 19:56:12.618916 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.618897    2580 scope.go:117] "RemoveContainer" containerID="e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"
Sep 18 19:56:12.619184 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:56:12.619163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba\": container with ID starting with e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba not found: ID does not exist" containerID="e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"
Sep 18 19:56:12.619227 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.619194    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba"} err="failed to get container status \"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba\": rpc error: code = NotFound desc = could not find container \"e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba\": container with ID starting with e92556f3c9f8e3a936c8a1707b032b6b3b98f42fb85e2926042460effc2b16ba not found: ID does not exist"
Sep 18 19:56:12.640090 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.640058    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:56:12.645141 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:12.645119    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesl5tf"]
Sep 18 19:56:13.294440 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:13.294394    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9456f745-4bf1-4785-953e-a83c67c49253" path="/var/lib/kubelet/pods/9456f745-4bf1-4785-953e-a83c67c49253/volumes"
Sep 18 19:56:14.233262 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:14.233209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:16.777217 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:16.777182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:16.962701 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:16.962647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:22.874128 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:22.874088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:22.995225 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:22.995190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:24.323680 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.323641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:24.432985 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.432943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:24.700923 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.700849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:56:24.701280 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main"
Sep 18 19:56:24.701338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main"
Sep 18 19:56:24.701338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701293    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main"
Sep 18 19:56:24.701338 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main"
Sep 18 19:56:24.701433 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701369    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9456f745-4bf1-4785-953e-a83c67c49253" containerName="main"
Sep 18 19:56:24.701433 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.701381    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d80d9f7-d7aa-42a1-9056-f3fc7d088fb3" containerName="main"
Sep 18 19:56:24.705688 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.705663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.711097 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.711071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wws8w\""
Sep 18 19:56:24.711097 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.711071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 19:56:24.711289 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.711143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 19:56:24.711289 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.711080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-tsh6r\""
Sep 18 19:56:24.711397 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.711326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 19:56:24.734205 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.734173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:56:24.777352 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.777312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:24.826438 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.826404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4db5f\" (UniqueName: \"kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.826614 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.826488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.927534 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.927497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.927726 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.927554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4db5f\" (UniqueName: \"kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.930023 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.930004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:24.954232 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:24.954166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4db5f\" (UniqueName: \"kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:25.015966 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.015936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:25.321932 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.321908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:56:25.324157 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:56:25.324126    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a120785_53ca_40df_b3cf_c7bf1a872f91.slice/crio-c8d1dfeebfb6716c47a01324f68ab8732ce256a73cb690b1eed2b27d6ece3d49 WatchSource:0}: Error finding container c8d1dfeebfb6716c47a01324f68ab8732ce256a73cb690b1eed2b27d6ece3d49: Status 404 returned error can't find the container with id c8d1dfeebfb6716c47a01324f68ab8732ce256a73cb690b1eed2b27d6ece3d49
Sep 18 19:56:25.676555 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.676466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" event={"ID":"9a120785-53ca-40df-b3cf-c7bf1a872f91","Type":"ContainerStarted","Data":"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"}
Sep 18 19:56:25.676555 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.676521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" event={"ID":"9a120785-53ca-40df-b3cf-c7bf1a872f91","Type":"ContainerStarted","Data":"c8d1dfeebfb6716c47a01324f68ab8732ce256a73cb690b1eed2b27d6ece3d49"}
Sep 18 19:56:25.676773 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.676593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:25.734279 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:25.734226    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podStartSLOduration=1.734210665 podStartE2EDuration="1.734210665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:25.733411114 +0000 UTC m=+2546.520691515" watchObservedRunningTime="2026-09-18 19:56:25.734210665 +0000 UTC m=+2546.521491089"
Sep 18 19:56:30.369185 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:30.369145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:31.186095 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:31.186059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:31.333854 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:31.333821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:36.219323 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:36.219289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:36.301732 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:36.301699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:39.719050 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:39.719009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:45.444736 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:45.444701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:45.763073 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:45.762988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:56.686024 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:56.685989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:56:57.653999 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:56:57.653960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:58.452417 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:57:58.452377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:59.690960 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:57:59.690920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:12.780340 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:12.780291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:12.810093 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:12.810061    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:58:12.880933 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:12.880900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:16.686541 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:16.686505    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:26.685949 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:26.685907    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:27.829739 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:27.829676    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" containerID="cri-o://ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39" gracePeriod=45
Sep 18 19:58:36.686287 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:36.686246    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:36.686788 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:36.686348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:58:46.686250 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:46.686173    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:50.122245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:50.122210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:50.225551 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:50.225521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:56.686189 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:56.686152    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:58.079646 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.079622    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:58:58.158345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.158263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs\") pod \"9a120785-53ca-40df-b3cf-c7bf1a872f91\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") "
Sep 18 19:58:58.158345 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.158313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4db5f\" (UniqueName: \"kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f\") pod \"9a120785-53ca-40df-b3cf-c7bf1a872f91\" (UID: \"9a120785-53ca-40df-b3cf-c7bf1a872f91\") "
Sep 18 19:58:58.160505 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.160477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f" (OuterVolumeSpecName: "kube-api-access-4db5f") pod "9a120785-53ca-40df-b3cf-c7bf1a872f91" (UID: "9a120785-53ca-40df-b3cf-c7bf1a872f91"). InnerVolumeSpecName "kube-api-access-4db5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:58.160505 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.160490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a120785-53ca-40df-b3cf-c7bf1a872f91" (UID: "9a120785-53ca-40df-b3cf-c7bf1a872f91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:58.259751 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.259722    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4db5f\" (UniqueName: \"kubernetes.io/projected/9a120785-53ca-40df-b3cf-c7bf1a872f91-kube-api-access-4db5f\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:58:58.259890 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.259750    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a120785-53ca-40df-b3cf-c7bf1a872f91-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 19:58:58.439245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.439160    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerID="ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39" exitCode=0
Sep 18 19:58:58.439245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.439219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"
Sep 18 19:58:58.439245 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.439229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" event={"ID":"9a120785-53ca-40df-b3cf-c7bf1a872f91","Type":"ContainerDied","Data":"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"}
Sep 18 19:58:58.439520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.439272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6" event={"ID":"9a120785-53ca-40df-b3cf-c7bf1a872f91","Type":"ContainerDied","Data":"c8d1dfeebfb6716c47a01324f68ab8732ce256a73cb690b1eed2b27d6ece3d49"}
Sep 18 19:58:58.439520 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.439294    2580 scope.go:117] "RemoveContainer" containerID="ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"
Sep 18 19:58:58.447982 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.447964    2580 scope.go:117] "RemoveContainer" containerID="ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"
Sep 18 19:58:58.448234 ip-10-0-138-2 kubenswrapper[2580]: E0918 19:58:58.448214    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39\": container with ID starting with ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39 not found: ID does not exist" containerID="ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"
Sep 18 19:58:58.448309 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.448242    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39"} err="failed to get container status \"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39\": rpc error: code = NotFound desc = could not find container \"ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39\": container with ID starting with ffbaf4741db2448030f58cbd141b52c8ef53774d7f9f67d11970446184489c39 not found: ID does not exist"
Sep 18 19:58:58.480830 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.480803    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:58:58.485812 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:58.485791    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schez5mx6"]
Sep 18 19:58:59.294443 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:58:59.294411    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" path="/var/lib/kubelet/pods/9a120785-53ca-40df-b3cf-c7bf1a872f91/volumes"
Sep 18 19:59:01.421365 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:01.421332    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:59:01.429042 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:01.429021    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 19:59:02.681109 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:02.681078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:02.778165 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:02.778134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:09.686561 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:09.686528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:13.576456 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.576419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:13.782291 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.782259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 19:59:13.782695 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.782680    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main"
Sep 18 19:59:13.782695 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.782696    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main"
Sep 18 19:59:13.782830 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.782776    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a120785-53ca-40df-b3cf-c7bf1a872f91" containerName="main"
Sep 18 19:59:13.787082 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.787060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:13.790442 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.790420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 19:59:13.790562 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.790538    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 19:59:13.792453 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.792432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 19:59:13.792593 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.792436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-f2j2n\""
Sep 18 19:59:13.792593 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.792493    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-zmfl5\""
Sep 18 19:59:13.811527 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.811506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 19:59:13.824491 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.824467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:13.900482 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.900402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:13.900656 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:13.900491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpfw\" (UniqueName: \"kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.001463 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.001432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.001626 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.001489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpfw\" (UniqueName: \"kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.003936 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.003915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.011437 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.011414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpfw\" (UniqueName: \"kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.097991 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.097969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.387861 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.387836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 19:59:14.389699 ip-10-0-138-2 kubenswrapper[2580]: W0918 19:59:14.389668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67909aa9_2332_42ca_8c12_0b33ddb96012.slice/crio-6ec10b55dd72216a7182c90f41ace67fcbffed24a63fcdaf69d38b9e2af1b832 WatchSource:0}: Error finding container 6ec10b55dd72216a7182c90f41ace67fcbffed24a63fcdaf69d38b9e2af1b832: Status 404 returned error can't find the container with id 6ec10b55dd72216a7182c90f41ace67fcbffed24a63fcdaf69d38b9e2af1b832
Sep 18 19:59:14.523655 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.523614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" event={"ID":"67909aa9-2332-42ca-8c12-0b33ddb96012","Type":"ContainerStarted","Data":"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"}
Sep 18 19:59:14.523843 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.523680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" event={"ID":"67909aa9-2332-42ca-8c12-0b33ddb96012","Type":"ContainerStarted","Data":"6ec10b55dd72216a7182c90f41ace67fcbffed24a63fcdaf69d38b9e2af1b832"}
Sep 18 19:59:14.523843 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.523709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:14.551046 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:14.550993    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podStartSLOduration=1.550978192 podStartE2EDuration="1.550978192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:14.549024531 +0000 UTC m=+2715.336304958" watchObservedRunningTime="2026-09-18 19:59:14.550978192 +0000 UTC m=+2715.338258596"
Sep 18 19:59:16.868639 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:16.868599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:16.965379 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:16.965329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:19.895960 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:19.895926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:34.221742 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:34.221705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:40.090303 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:40.090266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:40.183184 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:40.183147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:45.530399 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:45.530365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 19:59:46.817410 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:46.817344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:48.448163 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:48.448129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:48.727011 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:48.726918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:52.825218 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:52.825184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:52.965315 ip-10-0-138-2 kubenswrapper[2580]: I0918 19:59:52.965283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:06.905037 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:06.905000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:35.297654 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:35.297561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:35.300472 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:35.300430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 20:00:35.366956 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:35.366906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:35.530428 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:35.530388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:00:45.530449 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:45.530413    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:00:50.319786 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:50.319724    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" containerID="cri-o://2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250" gracePeriod=45
Sep 18 20:00:55.530144 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:55.530101    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:00:55.530617 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:00:55.530252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 20:01:05.530214 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:05.530171    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:15.530624 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:15.530538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:20.573030 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.573004    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 20:01:20.699780 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.699707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs\") pod \"67909aa9-2332-42ca-8c12-0b33ddb96012\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") "
Sep 18 20:01:20.699780 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.699750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdpfw\" (UniqueName: \"kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw\") pod \"67909aa9-2332-42ca-8c12-0b33ddb96012\" (UID: \"67909aa9-2332-42ca-8c12-0b33ddb96012\") "
Sep 18 20:01:20.701883 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.701853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw" (OuterVolumeSpecName: "kube-api-access-gdpfw") pod "67909aa9-2332-42ca-8c12-0b33ddb96012" (UID: "67909aa9-2332-42ca-8c12-0b33ddb96012"). InnerVolumeSpecName "kube-api-access-gdpfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:20.701883 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.701858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67909aa9-2332-42ca-8c12-0b33ddb96012" (UID: "67909aa9-2332-42ca-8c12-0b33ddb96012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:20.800907 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.800867    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67909aa9-2332-42ca-8c12-0b33ddb96012-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:01:20.800907 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:20.800903    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdpfw\" (UniqueName: \"kubernetes.io/projected/67909aa9-2332-42ca-8c12-0b33ddb96012-kube-api-access-gdpfw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:01:21.148423 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.148387    2580 generic.go:356] "Generic (PLEG): container finished" podID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerID="2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250" exitCode=0
Sep 18 20:01:21.148606 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.148438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"
Sep 18 20:01:21.148606 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.148479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" event={"ID":"67909aa9-2332-42ca-8c12-0b33ddb96012","Type":"ContainerDied","Data":"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"}
Sep 18 20:01:21.148606 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.148533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c" event={"ID":"67909aa9-2332-42ca-8c12-0b33ddb96012","Type":"ContainerDied","Data":"6ec10b55dd72216a7182c90f41ace67fcbffed24a63fcdaf69d38b9e2af1b832"}
Sep 18 20:01:21.148606 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.148560    2580 scope.go:117] "RemoveContainer" containerID="2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"
Sep 18 20:01:21.157506 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.157477    2580 scope.go:117] "RemoveContainer" containerID="2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"
Sep 18 20:01:21.157782 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:01:21.157761    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250\": container with ID starting with 2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250 not found: ID does not exist" containerID="2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"
Sep 18 20:01:21.157847 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.157789    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250"} err="failed to get container status \"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250\": rpc error: code = NotFound desc = could not find container \"2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250\": container with ID starting with 2f37a94b6c934ea17b32ab8a4a96c0e0ab21524a3e205dda968a20fe7232d250 not found: ID does not exist"
Sep 18 20:01:21.173817 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.173794    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 20:01:21.179001 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.178980    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schemnc4c"]
Sep 18 20:01:21.294957 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:21.294922    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" path="/var/lib/kubelet/pods/67909aa9-2332-42ca-8c12-0b33ddb96012/volumes"
Sep 18 20:01:25.346068 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:25.346038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:25.442157 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:25.442123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:31.587210 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:31.587177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:32.265372 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.265327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:32.464563 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.464530    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:01:32.464991 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.464975    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main"
Sep 18 20:01:32.464991 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.464991    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main"
Sep 18 20:01:32.465153 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.465072    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="67909aa9-2332-42ca-8c12-0b33ddb96012" containerName="main"
Sep 18 20:01:32.467983 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.467965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.475301 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.475283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-pwbb2\""
Sep 18 20:01:32.475404 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.475302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dc84j\""
Sep 18 20:01:32.475917 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.475901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:01:32.476016 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.475975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:01:32.476016 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.475998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:01:32.490367 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.490345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:01:32.493572 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.493548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4kxk\" (UniqueName: \"kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.493682 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.493602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.504509 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.504477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:32.594240 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.594208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4kxk\" (UniqueName: \"kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.594652 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.594252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.596482 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.596464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.604981 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.604961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4kxk\" (UniqueName: \"kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:32.778730 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:32.778696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:33.061973 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:33.061916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:01:33.064891 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:01:33.064868    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e27dbd2_28d2_48d0_a7ea_2ab207bfa2d1.slice/crio-08277f91a409e5613b65ea7786d64f35902acf27cd1f5a29c757cf41f2a6e15b WatchSource:0}: Error finding container 08277f91a409e5613b65ea7786d64f35902acf27cd1f5a29c757cf41f2a6e15b: Status 404 returned error can't find the container with id 08277f91a409e5613b65ea7786d64f35902acf27cd1f5a29c757cf41f2a6e15b
Sep 18 20:01:33.211520 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:33.211485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" event={"ID":"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1","Type":"ContainerStarted","Data":"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"}
Sep 18 20:01:33.211713 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:33.211552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" event={"ID":"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1","Type":"ContainerStarted","Data":"08277f91a409e5613b65ea7786d64f35902acf27cd1f5a29c757cf41f2a6e15b"}
Sep 18 20:01:33.211713 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:33.211602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:01:33.235651 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:33.235571    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podStartSLOduration=1.2355559409999999 podStartE2EDuration="1.235555941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:33.232902604 +0000 UTC m=+2854.020183008" watchObservedRunningTime="2026-09-18 20:01:33.235555941 +0000 UTC m=+2854.022836345"
Sep 18 20:01:37.454294 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:37.454255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:37.532275 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:37.532240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:45.297612 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:45.297510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:53.834875 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:53.834836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:53.935315 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:01:53.935282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:04.218294 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:04.218263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:02:43.870918 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:43.870878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:44.036646 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:44.036613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:44.357296 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:44.357262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:44.454807 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:44.454776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:47.852906 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:47.852872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:50.346327 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:50.346286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:55.239289 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:55.239251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:55.501385 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:55.501291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:55.601712 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:55.601677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:57.163211 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:57.163177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:57.240150 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:02:57.240113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:06.345400 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:06.345365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:34.425385 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:34.425353    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:03:34.438784 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:34.438744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:34.561106 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:34.561073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:44.218688 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:44.218651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:03:49.445401 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:49.445328    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main" containerID="cri-o://4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd" gracePeriod=45
Sep 18 20:03:51.392603 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.392551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:51.536697 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.536665    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:03:51.540470 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.540440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.543901 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.543879    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-rnh72\""
Sep 18 20:03:51.552699 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.552671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:03:51.567444 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.567419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:51.620247 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.620219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.620424 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.620370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v76v\" (UniqueName: \"kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.721640 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.721537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v76v\" (UniqueName: \"kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.721819 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.721642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.724287 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.724259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.731196 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.731172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v76v\" (UniqueName: \"kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:51.853489 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:51.853457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:52.130395 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:52.130369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:03:52.132685 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:03:52.132650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca4c9d9_0321_4064_afdc_8e4720741f64.slice/crio-d7acb3ee5412a15d0e40f823c8bac6d06cd4f3633f1274649e1b2840cb386021 WatchSource:0}: Error finding container d7acb3ee5412a15d0e40f823c8bac6d06cd4f3633f1274649e1b2840cb386021: Status 404 returned error can't find the container with id d7acb3ee5412a15d0e40f823c8bac6d06cd4f3633f1274649e1b2840cb386021
Sep 18 20:03:52.906135 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:52.906100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" event={"ID":"0ca4c9d9-0321-4064-afdc-8e4720741f64","Type":"ContainerStarted","Data":"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"}
Sep 18 20:03:52.906564 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:52.906159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" event={"ID":"0ca4c9d9-0321-4064-afdc-8e4720741f64","Type":"ContainerStarted","Data":"d7acb3ee5412a15d0e40f823c8bac6d06cd4f3633f1274649e1b2840cb386021"}
Sep 18 20:03:52.906564 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:52.906213    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:03:52.926692 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:52.926643    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podStartSLOduration=1.9266279050000001 podStartE2EDuration="1.926627905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:52.925908811 +0000 UTC m=+2993.713189231" watchObservedRunningTime="2026-09-18 20:03:52.926627905 +0000 UTC m=+2993.713908309"
Sep 18 20:03:54.218564 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:03:54.218528    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:01.451571 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:01.451544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:04:01.460110 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:01.460088    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:04:04.218022 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:04.217982    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:04.218456 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:04.218085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:04:07.861309 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:07.861268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.218259 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:14.218221    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:19.693119 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.693089    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:04:19.767098 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.767031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs\") pod \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") "
Sep 18 20:04:19.767227 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.767108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4kxk\" (UniqueName: \"kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk\") pod \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\" (UID: \"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1\") "
Sep 18 20:04:19.769255 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.769225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" (UID: "8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:19.769359 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.769258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk" (OuterVolumeSpecName: "kube-api-access-m4kxk") pod "8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" (UID: "8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1"). InnerVolumeSpecName "kube-api-access-m4kxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:19.868019 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.867987    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:04:19.868019 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:19.868013    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4kxk\" (UniqueName: \"kubernetes.io/projected/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1-kube-api-access-m4kxk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:04:20.038703 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.038618    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerID="4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd" exitCode=0
Sep 18 20:04:20.038703 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.038669    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"
Sep 18 20:04:20.038928 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.038698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" event={"ID":"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1","Type":"ContainerDied","Data":"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"}
Sep 18 20:04:20.038928 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.038752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw" event={"ID":"8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1","Type":"ContainerDied","Data":"08277f91a409e5613b65ea7786d64f35902acf27cd1f5a29c757cf41f2a6e15b"}
Sep 18 20:04:20.038928 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.038800    2580 scope.go:117] "RemoveContainer" containerID="4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"
Sep 18 20:04:20.047607 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.047573    2580 scope.go:117] "RemoveContainer" containerID="4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"
Sep 18 20:04:20.047868 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:04:20.047852    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd\": container with ID starting with 4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd not found: ID does not exist" containerID="4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"
Sep 18 20:04:20.047914 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.047875    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd"} err="failed to get container status \"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd\": rpc error: code = NotFound desc = could not find container \"4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd\": container with ID starting with 4edca1debe191a9c0df2ea152212f47e414cc85d1729140cfed241a6fd66d8cd not found: ID does not exist"
Sep 18 20:04:20.063313 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.063289    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:04:20.067005 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:20.066984    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-5m7zw"]
Sep 18 20:04:21.294661 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:21.294629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" path="/var/lib/kubelet/pods/8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1/volumes"
Sep 18 20:04:23.913013 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:04:23.912982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:05:00.309492 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:00.309447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:00.386618 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:00.386559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:01.250267 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:01.250239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:31.441546 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:31.441502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:31.510323 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:31.510283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:50.404347 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:50.404302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:50.497283 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:50.497249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:55.464126 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:05:55.464091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:02.963659 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:02.963616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:03.373006 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.372975    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:06:03.373394 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.373380    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main"
Sep 18 20:06:03.373444 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.373395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main"
Sep 18 20:06:03.373478 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.373464    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e27dbd2-28d2-48d0-a7ea-2ab207bfa2d1" containerName="main"
Sep 18 20:06:03.376518 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.376496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.381696 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.381663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-8jkbb\""
Sep 18 20:06:03.381841 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.381731    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 20:06:03.381841 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.381728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-jvfl6\""
Sep 18 20:06:03.381841 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.381729    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 20:06:03.382079 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.381850    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 20:06:03.391458 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.391431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:06:03.429341 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.429299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:03.484144 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.484110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.484316 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.484203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzpm\" (UniqueName: \"kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.585143 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.585105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.585364 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.585167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzpm\" (UniqueName: \"kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.587873 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.587839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.598021 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.597986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzpm\" (UniqueName: \"kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.687029 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.686930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:03.965952 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:03.965924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:06:03.967879 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:06:03.967856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438e9fac_1ba2_4ba8_bbcc_a6b53db74df9.slice/crio-de221001e0d1280c8926fab360cff83a62dec3636134a3e2b51de2b67ac8b581 WatchSource:0}: Error finding container de221001e0d1280c8926fab360cff83a62dec3636134a3e2b51de2b67ac8b581: Status 404 returned error can't find the container with id de221001e0d1280c8926fab360cff83a62dec3636134a3e2b51de2b67ac8b581
Sep 18 20:06:04.554944 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:04.554906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" event={"ID":"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9","Type":"ContainerStarted","Data":"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"}
Sep 18 20:06:04.555125 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:04.554958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" event={"ID":"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9","Type":"ContainerStarted","Data":"de221001e0d1280c8926fab360cff83a62dec3636134a3e2b51de2b67ac8b581"}
Sep 18 20:06:04.555125 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:04.555036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:04.579656 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:04.579567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podStartSLOduration=1.5795458390000001 podStartE2EDuration="1.579545839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:04.579041273 +0000 UTC m=+3125.366321677" watchObservedRunningTime="2026-09-18 20:06:04.579545839 +0000 UTC m=+3125.366826243"
Sep 18 20:06:11.062668 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:11.062630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:14.506173 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:14.506078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:14.545547 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:14.545515    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:06:14.599495 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:14.599459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:19.342510 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:19.342473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:19.997100 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:19.997007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:23.913068 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:23.913026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:26.089060 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:26.088998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:29.564768 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:29.564726    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main" containerID="cri-o://a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef" gracePeriod=45
Sep 18 20:06:32.799124 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:32.799089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:33.913160 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:33.913109    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:35.562156 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:35.562126    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:06:40.790136 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:40.790097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:42.111441 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:42.111406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:42.187123 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:42.187086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:43.912680 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:43.912642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:43.913129 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:43.912762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:06:53.912693 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:53.912653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:06:59.824865 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.824837    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:06:59.839209 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.839149    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerID="a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef" exitCode=0
Sep 18 20:06:59.839209 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.839187    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"
Sep 18 20:06:59.839356 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.839188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" event={"ID":"0ca4c9d9-0321-4064-afdc-8e4720741f64","Type":"ContainerDied","Data":"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"}
Sep 18 20:06:59.839356 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.839321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8" event={"ID":"0ca4c9d9-0321-4064-afdc-8e4720741f64","Type":"ContainerDied","Data":"d7acb3ee5412a15d0e40f823c8bac6d06cd4f3633f1274649e1b2840cb386021"}
Sep 18 20:06:59.839356 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.839351    2580 scope.go:117] "RemoveContainer" containerID="a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"
Sep 18 20:06:59.848940 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.848908    2580 scope.go:117] "RemoveContainer" containerID="a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"
Sep 18 20:06:59.849221 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:06:59.849201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef\": container with ID starting with a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef not found: ID does not exist" containerID="a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"
Sep 18 20:06:59.849306 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.849229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef"} err="failed to get container status \"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef\": rpc error: code = NotFound desc = could not find container \"a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef\": container with ID starting with a052a6671c326be023e2b0fa4adfb6d374cf19bf43706656ff2c9b7cee1c77ef not found: ID does not exist"
Sep 18 20:06:59.877650 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.877626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs\") pod \"0ca4c9d9-0321-4064-afdc-8e4720741f64\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") "
Sep 18 20:06:59.877751 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.877671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v76v\" (UniqueName: \"kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v\") pod \"0ca4c9d9-0321-4064-afdc-8e4720741f64\" (UID: \"0ca4c9d9-0321-4064-afdc-8e4720741f64\") "
Sep 18 20:06:59.879604 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.879568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ca4c9d9-0321-4064-afdc-8e4720741f64" (UID: "0ca4c9d9-0321-4064-afdc-8e4720741f64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:59.879726 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.879708    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v" (OuterVolumeSpecName: "kube-api-access-7v76v") pod "0ca4c9d9-0321-4064-afdc-8e4720741f64" (UID: "0ca4c9d9-0321-4064-afdc-8e4720741f64"). InnerVolumeSpecName "kube-api-access-7v76v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:59.978963 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.978935    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca4c9d9-0321-4064-afdc-8e4720741f64-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:06:59.978963 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:06:59.978958    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v76v\" (UniqueName: \"kubernetes.io/projected/0ca4c9d9-0321-4064-afdc-8e4720741f64-kube-api-access-7v76v\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:07:00.164668 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:00.164637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:07:00.170080 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:00.170056    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-flvj8"]
Sep 18 20:07:00.798845 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:00.798803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:00.899451 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:00.899398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:01.114028 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:01.113932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:01.295898 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:01.295868    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" path="/var/lib/kubelet/pods/0ca4c9d9-0321-4064-afdc-8e4720741f64/volumes"
Sep 18 20:07:03.982333 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:03.982297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:04.095902 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:04.095870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:04.204947 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:04.204911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:09.329312 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:09.329275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:11.357692 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:11.357655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:17.942800 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:17.942770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:18.018663 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:18.018632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:39.719273 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:07:39.719234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:52.565510 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:08:52.565474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:52.594629 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:08:52.594602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:08:53.831393 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:08:53.831361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:55.562251 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:08:55.562207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:08:57.854879 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:08:57.854849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:01.482691 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:01.482659    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:09:01.493312 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:01.493294    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:09:05.562615 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:05.562562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:09:07.612419 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:07.612377    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" containerID="cri-o://6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47" gracePeriod=45
Sep 18 20:09:15.562737 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:15.562654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:09:15.563190 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:15.562758    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:09:17.982980 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:17.982951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:18.164844 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:18.164810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:25.191719 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:25.191682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:25.387439 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:25.387409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:25.561733 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:25.561695    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:09:25.762391 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:25.762354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:35.561996 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:35.561960    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:09:37.867178 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:37.867154    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:09:38.043397 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.043315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs\") pod \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") "
Sep 18 20:09:38.043397 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.043372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqzpm\" (UniqueName: \"kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm\") pod \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\" (UID: \"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9\") "
Sep 18 20:09:38.045460 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.045432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" (UID: "438e9fac-1ba2-4ba8-bbcc-a6b53db74df9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:38.045590 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.045551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm" (OuterVolumeSpecName: "kube-api-access-rqzpm") pod "438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" (UID: "438e9fac-1ba2-4ba8-bbcc-a6b53db74df9"). InnerVolumeSpecName "kube-api-access-rqzpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:38.144639 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.144603    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzpm\" (UniqueName: \"kubernetes.io/projected/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-kube-api-access-rqzpm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:09:38.144639 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.144632    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:09:38.616844 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.616814    2580 generic.go:356] "Generic (PLEG): container finished" podID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerID="6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47" exitCode=0
Sep 18 20:09:38.617031 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.616858    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"
Sep 18 20:09:38.617031 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.616907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" event={"ID":"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9","Type":"ContainerDied","Data":"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"}
Sep 18 20:09:38.617031 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.616964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt" event={"ID":"438e9fac-1ba2-4ba8-bbcc-a6b53db74df9","Type":"ContainerDied","Data":"de221001e0d1280c8926fab360cff83a62dec3636134a3e2b51de2b67ac8b581"}
Sep 18 20:09:38.617031 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.616989    2580 scope.go:117] "RemoveContainer" containerID="6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"
Sep 18 20:09:38.625668 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.625639    2580 scope.go:117] "RemoveContainer" containerID="6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"
Sep 18 20:09:38.625941 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:09:38.625917    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47\": container with ID starting with 6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47 not found: ID does not exist" containerID="6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"
Sep 18 20:09:38.626022 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.625949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47"} err="failed to get container status \"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47\": rpc error: code = NotFound desc = could not find container \"6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47\": container with ID starting with 6c113b551356f27430585025f59993044651393aa1f02c89dc6df78fbf634e47 not found: ID does not exist"
Sep 18 20:09:38.647128 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.647108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:09:38.653211 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:38.653189    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebwsxt"]
Sep 18 20:09:39.294057 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:39.294020    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" path="/var/lib/kubelet/pods/438e9fac-1ba2-4ba8-bbcc-a6b53db74df9/volumes"
Sep 18 20:09:43.237683 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:43.237638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:43.324812 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:43.324780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:49.635122 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:49.635087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:57.151033 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:57.150987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:57.324857 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:09:57.324826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:02.426041 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.426004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:02.698557 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.698474    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:10:02.698956 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.698938    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main"
Sep 18 20:10:02.699047 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.698959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main"
Sep 18 20:10:02.699047 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.698980    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main"
Sep 18 20:10:02.699047 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.698988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main"
Sep 18 20:10:02.699196 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.699090    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e9fac-1ba2-4ba8-bbcc-a6b53db74df9" containerName="main"
Sep 18 20:10:02.699196 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.699103    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca4c9d9-0321-4064-afdc-8e4720741f64" containerName="main"
Sep 18 20:10:02.703233 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.703211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.709052 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.709031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xz8hh\""
Sep 18 20:10:02.709883 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.709862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-w2flf\""
Sep 18 20:10:02.710006 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.709883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:10:02.710006 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.709934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:10:02.710006 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.709863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:10:02.722042 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.722017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:10:02.730697 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.730666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:02.733475 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.733453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.733562 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.733501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9dn\" (UniqueName: \"kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.834238 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.834206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.834238 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.834249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9dn\" (UniqueName: \"kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.836833 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.836801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:02.852768 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:02.852743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9dn\" (UniqueName: \"kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:03.013127 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.013036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:03.297393 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.297370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:10:03.299679 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:10:03.299650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a3af4_0f9a_427a_9de0_0f315f22a544.slice/crio-884d30ca81d6a686bf93dc78d5875806118167c480b5c8528f407c381d33ee67 WatchSource:0}: Error finding container 884d30ca81d6a686bf93dc78d5875806118167c480b5c8528f407c381d33ee67: Status 404 returned error can't find the container with id 884d30ca81d6a686bf93dc78d5875806118167c480b5c8528f407c381d33ee67
Sep 18 20:10:03.740814 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.740720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" event={"ID":"486a3af4-0f9a-427a-9de0-0f315f22a544","Type":"ContainerStarted","Data":"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"}
Sep 18 20:10:03.740814 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.740778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" event={"ID":"486a3af4-0f9a-427a-9de0-0f315f22a544","Type":"ContainerStarted","Data":"884d30ca81d6a686bf93dc78d5875806118167c480b5c8528f407c381d33ee67"}
Sep 18 20:10:03.741305 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.740844    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:10:03.769784 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:03.769733    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podStartSLOduration=1.769717849 podStartE2EDuration="1.769717849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:03.768376596 +0000 UTC m=+3364.555657000" watchObservedRunningTime="2026-09-18 20:10:03.769717849 +0000 UTC m=+3364.556998286"
Sep 18 20:10:14.680705 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:14.680671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:34.747662 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:10:34.747631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:11:14.413601 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:14.413541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:19.222815 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:19.222772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:34.578527 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:34.578485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:34.686399 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:34.686366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:34.830843 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:34.830775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:35.055363 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:35.055313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:35.153532 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:35.153445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:37.492855 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:37.492821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:37.577378 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:37.577341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.226625 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:38.226596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.194033 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:43.193999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.365054 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:11:43.365020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:02.243434 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:02.243395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:21.128955 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:21.128866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:21.140639 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:21.140607    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:12:21.206741 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:21.206708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:24.747963 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:24.747925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:12:34.747869 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:34.747829    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:12:36.159788 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:36.159712    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" containerID="cri-o://7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de" gracePeriod=45
Sep 18 20:12:44.747863 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:44.747820    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:12:44.748243 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:44.747932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:12:54.747828 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:12:54.747791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:04.747838 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:04.747797    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:06.406862 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.406837    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:13:06.483132 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.483056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs\") pod \"486a3af4-0f9a-427a-9de0-0f315f22a544\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") "
Sep 18 20:13:06.483132 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.483100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc9dn\" (UniqueName: \"kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn\") pod \"486a3af4-0f9a-427a-9de0-0f315f22a544\" (UID: \"486a3af4-0f9a-427a-9de0-0f315f22a544\") "
Sep 18 20:13:06.485272 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.485246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "486a3af4-0f9a-427a-9de0-0f315f22a544" (UID: "486a3af4-0f9a-427a-9de0-0f315f22a544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:06.485378 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.485289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn" (OuterVolumeSpecName: "kube-api-access-vc9dn") pod "486a3af4-0f9a-427a-9de0-0f315f22a544" (UID: "486a3af4-0f9a-427a-9de0-0f315f22a544"). InnerVolumeSpecName "kube-api-access-vc9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:06.584085 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.584054    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a3af4-0f9a-427a-9de0-0f315f22a544-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:13:06.584085 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.584080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9dn\" (UniqueName: \"kubernetes.io/projected/486a3af4-0f9a-427a-9de0-0f315f22a544-kube-api-access-vc9dn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:13:06.629250 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.629218    2580 generic.go:356] "Generic (PLEG): container finished" podID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerID="7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de" exitCode=0
Sep 18 20:13:06.629426 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.629276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"
Sep 18 20:13:06.629426 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.629306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" event={"ID":"486a3af4-0f9a-427a-9de0-0f315f22a544","Type":"ContainerDied","Data":"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"}
Sep 18 20:13:06.629426 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.629402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2" event={"ID":"486a3af4-0f9a-427a-9de0-0f315f22a544","Type":"ContainerDied","Data":"884d30ca81d6a686bf93dc78d5875806118167c480b5c8528f407c381d33ee67"}
Sep 18 20:13:06.629553 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.629430    2580 scope.go:117] "RemoveContainer" containerID="7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"
Sep 18 20:13:06.638722 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.638706    2580 scope.go:117] "RemoveContainer" containerID="7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"
Sep 18 20:13:06.638951 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:13:06.638934    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de\": container with ID starting with 7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de not found: ID does not exist" containerID="7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"
Sep 18 20:13:06.638996 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.638956    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de"} err="failed to get container status \"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de\": rpc error: code = NotFound desc = could not find container \"7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de\": container with ID starting with 7b0cecf0464f397181b897bdd01d3eb82781d79141be5277760e27bf0bd259de not found: ID does not exist"
Sep 18 20:13:06.657901 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.657868    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:13:06.660883 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:06.660859    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48dgdb2"]
Sep 18 20:13:07.294966 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:07.294937    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" path="/var/lib/kubelet/pods/486a3af4-0f9a-427a-9de0-0f315f22a544/volumes"
Sep 18 20:13:12.996338 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:12.996285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.219739 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:13.219694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.432503 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:13.432468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.551749 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:13.551714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:17.120966 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:17.120929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:19.419341 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:19.419302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:24.401928 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.401897    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:13:24.402269 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.402256    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main"
Sep 18 20:13:24.402325 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.402270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main"
Sep 18 20:13:24.402364 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.402340    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="486a3af4-0f9a-427a-9de0-0f315f22a544" containerName="main"
Sep 18 20:13:24.406947 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.406929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.411475 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411454    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:13:24.411618 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411554    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 20:13:24.411700 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:13:24.411949 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lncpr\""
Sep 18 20:13:24.412035 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:13:24.412035 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.411932    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:13:24.429713 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.429686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:13:24.486185 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.486156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:24.534066 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534204 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534204 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534204 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534204 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcn5h\" (UniqueName: \"kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534339 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534339 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.534339 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.534269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635660 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635660 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcn5h\" (UniqueName: \"kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.635899 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.635863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.636201 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.636079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.636201 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.636128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.636201 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.636153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.636372 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.636234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.636529 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.636509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.638128 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.638108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.638367 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.638352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.647220 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.647196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcn5h\" (UniqueName: \"kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h\") pod \"router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:24.717386 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:24.717298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:13:25.014322 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:25.014246    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:13:25.026655 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:13:25.026627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b553d04_c9ec_4b66_b82b_6653072f8474.slice/crio-d76b17d04576009711b0ccd65251e6aa1c4566c6fefd559b2f9f38fc05eefd6c WatchSource:0}: Error finding container d76b17d04576009711b0ccd65251e6aa1c4566c6fefd559b2f9f38fc05eefd6c: Status 404 returned error can't find the container with id d76b17d04576009711b0ccd65251e6aa1c4566c6fefd559b2f9f38fc05eefd6c
Sep 18 20:13:25.195965 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:25.195931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:25.293399 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:25.293292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:25.725008 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:25.724967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerStarted","Data":"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"}
Sep 18 20:13:25.725395 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:25.725038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerStarted","Data":"d76b17d04576009711b0ccd65251e6aa1c4566c6fefd559b2f9f38fc05eefd6c"}
Sep 18 20:13:26.728887 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:26.728796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:26.883933 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:26.883895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:27.738528 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:27.738495    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerID="17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea" exitCode=0
Sep 18 20:13:27.738934 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:27.738549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerDied","Data":"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"}
Sep 18 20:13:27.739862 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:27.739848    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:13:54.910889 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:54.910797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerStarted","Data":"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"}
Sep 18 20:13:54.946968 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:54.946918    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podStartSLOduration=1.954774137 podStartE2EDuration="30.946901392s" totalImagesPullingTime="26.819894874s" totalInitContainerRuntime="2.172232381s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:54.944077802 +0000 UTC m=+3595.731358220" watchObservedRunningTime="2026-09-18 20:13:54.946901392 +0000 UTC m=+3595.734181819"
Sep 18 20:13:56.679820 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:13:56.679779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:01.515063 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:01.515033    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:14:01.526118 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:01.526097    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:14:04.718026 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:04.717989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:14:04.718452 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:04.718043    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:14:04.719571 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:04.719539    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:14:14.718364 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:14.718316    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:14:16.774577 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:16.774539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:16.947654 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:16.947617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:17.110897 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:17.110860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:17.423988 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:17.423869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:17.580793 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:17.580760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.333354 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:19.333315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.461395 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:19.461356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:20.516909 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:20.516874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:23.187897 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:23.187864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:23.290888 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:23.290848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:23.489884 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:23.489804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:23.574173 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:23.574139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:24.035403 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:24.035369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:24.717965 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:24.717928    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:14:30.563803 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:30.563765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:30.665474 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:30.665436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:34.718767 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:34.718712    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:14:40.126269 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:40.126237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:43.159203 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:43.159164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:44.718308 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:44.718258    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:14:54.718244 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:14:54.718199    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:15:04.718686 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:04.718640    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:15:05.394031 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:05.393994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:14.718284 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:14.718195    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:15:24.718044 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:24.718000    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:15:33.878916 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:33.878879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:34.718437 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:34.718395    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 20:15:44.727792 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:44.727755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:15:44.735487 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:44.735460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:15:53.460709 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:53.460674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:15:53.463778 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:53.463748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:53.537405 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:15:53.537375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:08.485409 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:08.485316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main" containerID="cri-o://67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b" gracePeriod=45
Sep 18 20:16:09.128768 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.128746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:16:09.224160 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224160 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224133    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224160 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224383 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224383 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcn5h\" (UniqueName: \"kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224383 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224267    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224383 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224383 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm\") pod \"3b553d04-c9ec-4b66-b82b-6653072f8474\" (UID: \"3b553d04-c9ec-4b66-b82b-6653072f8474\") "
Sep 18 20:16:09.224574 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:09.224845 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.224809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.225108 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.225087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home" (OuterVolumeSpecName: "home") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.226769 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.226739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h" (OuterVolumeSpecName: "kube-api-access-dcn5h") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "kube-api-access-dcn5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:09.226870 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.226781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm" (OuterVolumeSpecName: "dshm") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.226922 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.226878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:09.241430 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.241402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.246291 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.246269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b553d04-c9ec-4b66-b82b-6653072f8474" (UID: "3b553d04-c9ec-4b66-b82b-6653072f8474"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:09.325268 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325236    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325268 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325264    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b553d04-c9ec-4b66-b82b-6653072f8474-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325276    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-kserve-provision-location\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325291    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b553d04-c9ec-4b66-b82b-6653072f8474-cabundle-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325303    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcn5h\" (UniqueName: \"kubernetes.io/projected/3b553d04-c9ec-4b66-b82b-6653072f8474-kube-api-access-dcn5h\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325313    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325331    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.325427 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.325342    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b553d04-c9ec-4b66-b82b-6653072f8474-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:16:09.591682 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.591652    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerID="67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b" exitCode=0
Sep 18 20:16:09.592061 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.591700    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"
Sep 18 20:16:09.592061 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.591716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerDied","Data":"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"}
Sep 18 20:16:09.592061 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.591757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68" event={"ID":"3b553d04-c9ec-4b66-b82b-6653072f8474","Type":"ContainerDied","Data":"d76b17d04576009711b0ccd65251e6aa1c4566c6fefd559b2f9f38fc05eefd6c"}
Sep 18 20:16:09.592061 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.591777    2580 scope.go:117] "RemoveContainer" containerID="67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"
Sep 18 20:16:09.600261 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.600244    2580 scope.go:117] "RemoveContainer" containerID="17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"
Sep 18 20:16:09.607754 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.607738    2580 scope.go:117] "RemoveContainer" containerID="67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"
Sep 18 20:16:09.608008 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:16:09.607988    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b\": container with ID starting with 67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b not found: ID does not exist" containerID="67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"
Sep 18 20:16:09.608073 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.608014    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b"} err="failed to get container status \"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b\": rpc error: code = NotFound desc = could not find container \"67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b\": container with ID starting with 67a7c1bb7f44942b3d12ad49ee5de91a1eb60c6eb92a7ed298666dca43c8d29b not found: ID does not exist"
Sep 18 20:16:09.608073 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.608027    2580 scope.go:117] "RemoveContainer" containerID="17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"
Sep 18 20:16:09.608274 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:16:09.608255    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea\": container with ID starting with 17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea not found: ID does not exist" containerID="17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"
Sep 18 20:16:09.608314 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.608279    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea"} err="failed to get container status \"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea\": rpc error: code = NotFound desc = could not find container \"17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea\": container with ID starting with 17935c2ef320db39279e5fd4dda46d8c0e301d99b1e545a0200409accd7b1dea not found: ID does not exist"
Sep 18 20:16:09.613962 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.613938    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:16:09.617644 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:09.617622    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-77577f4cd9-g5q68"]
Sep 18 20:16:11.295665 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:11.295630    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" path="/var/lib/kubelet/pods/3b553d04-c9ec-4b66-b82b-6653072f8474/volumes"
Sep 18 20:16:43.532567 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:43.532465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:43.639902 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:43.639868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:48.587426 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:48.587392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:58.224320 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:16:58.224275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:09.806178 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:09.806143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:09.950968 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:09.950930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.883073 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:11.883037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:18.717090 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:18.717057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:21.690113 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:21.690076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:30.789244 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:30.789213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:30.891733 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:30.891692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:30.990486 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:30.990456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:31.262300 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:31.262262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:34.775829 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:34.775795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:34.851889 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:34.851853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:36.911110 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:36.911079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:36.985256 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:36.985229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:37.245744 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:37.245661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:37.512997 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:37.512919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:38.091901 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:38.091872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:38.187391 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:38.187359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:39.044124 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:39.044085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:48.198983 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:48.198947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:48.397831 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:48.397793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:50.646822 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:17:50.646789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:01.543694 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:01.543664    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:19:01.555258 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:01.555241    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:19:02.768422 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:02.768368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.808780 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:22.808742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.905561 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:22.905513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:29.631529 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:29.631497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:34.748403 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:34.748367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:35.024985 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.024867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:35.145891 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.145838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:35.315436 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315406    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:19:35.315797 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315785    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="storage-initializer"
Sep 18 20:19:35.315847 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315799    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="storage-initializer"
Sep 18 20:19:35.315847 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315816    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main"
Sep 18 20:19:35.315847 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main"
Sep 18 20:19:35.315945 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.315881    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b553d04-c9ec-4b66-b82b-6653072f8474" containerName="main"
Sep 18 20:19:35.318946 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.318930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.323775 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.323750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 20:19:35.325825 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.325518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 20:19:35.325825 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.325557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 20:19:35.325825 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.325518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-2sxg9\""
Sep 18 20:19:35.325825 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.325715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-tklqs\""
Sep 18 20:19:35.389932 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.389895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:19:35.413410 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.413384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:35.450881 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.450852    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:19:35.452368 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.452345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lbm\" (UniqueName: \"kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.452444 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.452411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.454406 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.454391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.457695 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.457674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 20:19:35.457804 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.457731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j5f7g\""
Sep 18 20:19:35.457804 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.457684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:19:35.473402 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.473381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:19:35.553130 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lbm\" (UniqueName: \"kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.553295 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmcq\" (UniqueName: \"kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553295 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553295 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553421 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.553421 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553421 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553516 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553516 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.553610 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.553522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.555819 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.555796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.562438 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.562421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lbm\" (UniqueName: \"kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.630758 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.630687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:35.654231 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654371 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654371 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654371 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654371 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654598 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmcq\" (UniqueName: \"kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654598 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654598 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654753 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.654874 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.655020 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.654999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.655106 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.655087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.655150 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.655099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.656432 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.656413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.656757 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.656739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.664455 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.664426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmcq\" (UniqueName: \"kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.764790 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.764754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:19:35.905445 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:35.905284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:19:35.908091 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:19:35.908069    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44e6941_aec3_498f_b4a3_b1f73869954b.slice/crio-8d88d16bd3bfc2ce2f4156c2740442477537d9f94483db7ba8a941af834f0197 WatchSource:0}: Error finding container 8d88d16bd3bfc2ce2f4156c2740442477537d9f94483db7ba8a941af834f0197: Status 404 returned error can't find the container with id 8d88d16bd3bfc2ce2f4156c2740442477537d9f94483db7ba8a941af834f0197
Sep 18 20:19:36.055694 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.055665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:19:36.057049 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:19:36.057027    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b20f041_6a98_468e_88d3_1c928c05f4bc.slice/crio-cbfdecf624b02caae7266c8058870f036bec74955de56bd99c83739326e543e7 WatchSource:0}: Error finding container cbfdecf624b02caae7266c8058870f036bec74955de56bd99c83739326e543e7: Status 404 returned error can't find the container with id cbfdecf624b02caae7266c8058870f036bec74955de56bd99c83739326e543e7
Sep 18 20:19:36.573101 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.573067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerStarted","Data":"ed8cbe31527c74508e6311f1695a4536da266416ac7fa0f91b3f9de3cad17cf0"}
Sep 18 20:19:36.573279 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.573124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerStarted","Data":"cbfdecf624b02caae7266c8058870f036bec74955de56bd99c83739326e543e7"}
Sep 18 20:19:36.574753 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.574728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" event={"ID":"c44e6941-aec3-498f-b4a3-b1f73869954b","Type":"ContainerStarted","Data":"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"}
Sep 18 20:19:36.574856 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.574768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" event={"ID":"c44e6941-aec3-498f-b4a3-b1f73869954b","Type":"ContainerStarted","Data":"8d88d16bd3bfc2ce2f4156c2740442477537d9f94483db7ba8a941af834f0197"}
Sep 18 20:19:36.574896 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.574856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:19:36.633514 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:36.633464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podStartSLOduration=1.633449187 podStartE2EDuration="1.633449187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:36.630818342 +0000 UTC m=+3937.418098745" watchObservedRunningTime="2026-09-18 20:19:36.633449187 +0000 UTC m=+3937.420729589"
Sep 18 20:19:37.435050 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:37.435011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:37.516981 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:37.516943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:38.586294 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:38.586258    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerID="ed8cbe31527c74508e6311f1695a4536da266416ac7fa0f91b3f9de3cad17cf0" exitCode=0
Sep 18 20:19:38.586703 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:38.586336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerDied","Data":"ed8cbe31527c74508e6311f1695a4536da266416ac7fa0f91b3f9de3cad17cf0"}
Sep 18 20:19:39.593605 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:39.593551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerStarted","Data":"8e9dbff5b128fce99cbbf7d9bda942b516a4f01f2907bcef85518015af328720"}
Sep 18 20:19:39.622011 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:39.621953    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.47200715 podStartE2EDuration="4.621936937s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149929787s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:39.620222512 +0000 UTC m=+3940.407502914" watchObservedRunningTime="2026-09-18 20:19:39.621936937 +0000 UTC m=+3940.409217339"
Sep 18 20:19:53.762934 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:53.762826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:59.661374 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:59.661335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:59.733209 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:19:59.733166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:07.582224 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:20:07.582194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:20:49.718791 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:20:49.718724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:49.831468 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:20:49.831408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:56.508575 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:20:56.508534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.484608 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:01.484547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.759035 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:01.758907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:03.446568 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:03.446525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:03.524924 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:03.524887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:11.293236 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:11.293194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.165322 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:17.165215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.270696 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:17.270659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.326325 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:21.326272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.349050 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:21.349023    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:21:21.405001 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:21.404955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:27.582494 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:27.582452    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:36.367870 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:36.367808    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main" containerID="cri-o://122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179" gracePeriod=45
Sep 18 20:21:37.581540 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:37.581502    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:37.735504 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:37.735465    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:21:47.581718 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:47.581676    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:47.582158 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:47.581766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:21:52.756994 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:52.756929    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="main" containerID="cri-o://8e9dbff5b128fce99cbbf7d9bda942b516a4f01f2907bcef85518015af328720" gracePeriod=45
Sep 18 20:21:53.286318 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.286288    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerID="8e9dbff5b128fce99cbbf7d9bda942b516a4f01f2907bcef85518015af328720" exitCode=0
Sep 18 20:21:53.286444 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.286360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerDied","Data":"8e9dbff5b128fce99cbbf7d9bda942b516a4f01f2907bcef85518015af328720"}
Sep 18 20:21:53.421014 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.420988    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:21:53.470147 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470117    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470328 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470328 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470328 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470328 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470552 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470552 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcmcq\" (UniqueName: \"kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470552 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs\") pod \"8b20f041-6a98-468e-88d3-1c928c05f4bc\" (UID: \"8b20f041-6a98-468e-88d3-1c928c05f4bc\") "
Sep 18 20:21:53.470552 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.470792 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:53.470978 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470933    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.470978 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.470955    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b20f041-6a98-468e-88d3-1c928c05f4bc-cabundle-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.471174 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.471147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home" (OuterVolumeSpecName: "home") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.472986 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.472961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:53.473322 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.473300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm" (OuterVolumeSpecName: "dshm") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.473493 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.473474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq" (OuterVolumeSpecName: "kube-api-access-pcmcq") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "kube-api-access-pcmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:53.482090 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.482065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.493377 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.493304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b20f041-6a98-468e-88d3-1c928c05f4bc" (UID: "8b20f041-6a98-468e-88d3-1c928c05f4bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.572111 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572075    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.572111 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572104    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-kserve-provision-location\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.572111 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572115    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.572338 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572124    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20f041-6a98-468e-88d3-1c928c05f4bc-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.572338 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572131    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmcq\" (UniqueName: \"kubernetes.io/projected/8b20f041-6a98-468e-88d3-1c928c05f4bc-kube-api-access-pcmcq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.572338 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:53.572139    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20f041-6a98-468e-88d3-1c928c05f4bc-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:21:54.292594 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.292563    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 20:21:54.292996 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.292605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8b20f041-6a98-468e-88d3-1c928c05f4bc","Type":"ContainerDied","Data":"cbfdecf624b02caae7266c8058870f036bec74955de56bd99c83739326e543e7"}
Sep 18 20:21:54.292996 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.292652    2580 scope.go:117] "RemoveContainer" containerID="8e9dbff5b128fce99cbbf7d9bda942b516a4f01f2907bcef85518015af328720"
Sep 18 20:21:54.301743 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.301725    2580 scope.go:117] "RemoveContainer" containerID="ed8cbe31527c74508e6311f1695a4536da266416ac7fa0f91b3f9de3cad17cf0"
Sep 18 20:21:54.333205 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.333180    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:21:54.342492 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:54.342469    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 20:21:55.294023 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:55.293988    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" path="/var/lib/kubelet/pods/8b20f041-6a98-468e-88d3-1c928c05f4bc/volumes"
Sep 18 20:21:57.582149 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:21:57.582099    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:22:06.620139 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.620111    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:22:06.684094 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.684026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6lbm\" (UniqueName: \"kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm\") pod \"c44e6941-aec3-498f-b4a3-b1f73869954b\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") "
Sep 18 20:22:06.684240 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.684109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs\") pod \"c44e6941-aec3-498f-b4a3-b1f73869954b\" (UID: \"c44e6941-aec3-498f-b4a3-b1f73869954b\") "
Sep 18 20:22:06.686274 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.686241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm" (OuterVolumeSpecName: "kube-api-access-q6lbm") pod "c44e6941-aec3-498f-b4a3-b1f73869954b" (UID: "c44e6941-aec3-498f-b4a3-b1f73869954b"). InnerVolumeSpecName "kube-api-access-q6lbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:06.686385 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.686320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c44e6941-aec3-498f-b4a3-b1f73869954b" (UID: "c44e6941-aec3-498f-b4a3-b1f73869954b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:06.784742 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.784712    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6lbm\" (UniqueName: \"kubernetes.io/projected/c44e6941-aec3-498f-b4a3-b1f73869954b-kube-api-access-q6lbm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.784742 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:06.784737    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44e6941-aec3-498f-b4a3-b1f73869954b-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:22:07.355942 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.355907    2580 generic.go:356] "Generic (PLEG): container finished" podID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerID="122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179" exitCode=0
Sep 18 20:22:07.356118 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.355952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"
Sep 18 20:22:07.356118 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.355987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" event={"ID":"c44e6941-aec3-498f-b4a3-b1f73869954b","Type":"ContainerDied","Data":"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"}
Sep 18 20:22:07.356118 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.356042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4" event={"ID":"c44e6941-aec3-498f-b4a3-b1f73869954b","Type":"ContainerDied","Data":"8d88d16bd3bfc2ce2f4156c2740442477537d9f94483db7ba8a941af834f0197"}
Sep 18 20:22:07.356118 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.356072    2580 scope.go:117] "RemoveContainer" containerID="122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"
Sep 18 20:22:07.364307 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.364293    2580 scope.go:117] "RemoveContainer" containerID="122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"
Sep 18 20:22:07.364562 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:22:07.364546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179\": container with ID starting with 122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179 not found: ID does not exist" containerID="122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"
Sep 18 20:22:07.364710 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.364572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179"} err="failed to get container status \"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179\": rpc error: code = NotFound desc = could not find container \"122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179\": container with ID starting with 122f8ddae2e1178a10097060e1e9cd02f1973f4ff24e6dbe205da9824610b179 not found: ID does not exist"
Sep 18 20:22:07.379447 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.379427    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:22:07.382786 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:07.382766    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schetqrn4"]
Sep 18 20:22:09.294936 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:09.294893    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" path="/var/lib/kubelet/pods/c44e6941-aec3-498f-b4a3-b1f73869954b/volumes"
Sep 18 20:22:11.423157 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:11.423120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:11.519784 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:11.519750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:17.497740 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.497699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:17.675937 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.675895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:22:17.676464 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main"
Sep 18 20:22:17.676539 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676467    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main"
Sep 18 20:22:17.676539 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="main"
Sep 18 20:22:17.676539 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676486    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="main"
Sep 18 20:22:17.676539 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676503    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="storage-initializer"
Sep 18 20:22:17.676539 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676512    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="storage-initializer"
Sep 18 20:22:17.676929 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676633    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44e6941-aec3-498f-b4a3-b1f73869954b" containerName="main"
Sep 18 20:22:17.676929 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.676646    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b20f041-6a98-468e-88d3-1c928c05f4bc" containerName="main"
Sep 18 20:22:17.679976 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.679946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.685641 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.685614    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 20:22:17.686002 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.685984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 20:22:17.686203 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.686178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-82nbc\""
Sep 18 20:22:17.686288 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.686192    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 20:22:17.686910 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.686486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-xls4b\""
Sep 18 20:22:17.689665 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.689640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:22:17.707608 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.707575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:17.776826 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.776750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.776826 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.776788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v745x\" (UniqueName: \"kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.877832 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.877799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.877832 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.877836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v745x\" (UniqueName: \"kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.880366 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.880335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.888032 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.888009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v745x\" (UniqueName: \"kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:17.992276 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:17.992246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:18.275638 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:18.275612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:22:18.281464 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:22:18.281435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a10b27_221f_4153_ae74_132fdf500a63.slice/crio-7b26cff63cfa2fffc2abe7edbb55b03600c17e6d23fbbb90d54d0b7448493cd3 WatchSource:0}: Error finding container 7b26cff63cfa2fffc2abe7edbb55b03600c17e6d23fbbb90d54d0b7448493cd3: Status 404 returned error can't find the container with id 7b26cff63cfa2fffc2abe7edbb55b03600c17e6d23fbbb90d54d0b7448493cd3
Sep 18 20:22:18.411998 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:18.411965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" event={"ID":"87a10b27-221f-4153-ae74-132fdf500a63","Type":"ContainerStarted","Data":"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"}
Sep 18 20:22:18.412255 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:18.412017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" event={"ID":"87a10b27-221f-4153-ae74-132fdf500a63","Type":"ContainerStarted","Data":"7b26cff63cfa2fffc2abe7edbb55b03600c17e6d23fbbb90d54d0b7448493cd3"}
Sep 18 20:22:18.412255 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:18.412160    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:18.448848 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:18.448798    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podStartSLOduration=1.44878178 podStartE2EDuration="1.44878178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:18.447456013 +0000 UTC m=+4099.234736429" watchObservedRunningTime="2026-09-18 20:22:18.44878178 +0000 UTC m=+4099.236062207"
Sep 18 20:22:19.384165 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:19.384132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:25.286614 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:25.286559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:25.582545 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:25.582515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:25.661733 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:25.661693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:49.418548 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:49.418466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:22:51.325616 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:51.325565    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:22:51.329489 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:51.329458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:51.443879 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:51.443842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:59.418966 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:22:59.418927    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:06.345523 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:06.345478    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main" containerID="cri-o://0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c" gracePeriod=45
Sep 18 20:23:09.418028 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:09.417987    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:14.955364 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:14.955329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:15.056173 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:15.056143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:15.212493 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:15.212410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:15.300745 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:15.300715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:19.418095 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:19.418052    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:19.418464 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:19.418154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:23:29.418307 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:29.418268    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:32.468625 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:32.468594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:36.595171 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.595143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:23:36.657937 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.657850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v745x\" (UniqueName: \"kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x\") pod \"87a10b27-221f-4153-ae74-132fdf500a63\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") "
Sep 18 20:23:36.657937 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.657923    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs\") pod \"87a10b27-221f-4153-ae74-132fdf500a63\" (UID: \"87a10b27-221f-4153-ae74-132fdf500a63\") "
Sep 18 20:23:36.660070 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.660040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87a10b27-221f-4153-ae74-132fdf500a63" (UID: "87a10b27-221f-4153-ae74-132fdf500a63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:36.660181 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.660068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x" (OuterVolumeSpecName: "kube-api-access-v745x") pod "87a10b27-221f-4153-ae74-132fdf500a63" (UID: "87a10b27-221f-4153-ae74-132fdf500a63"). InnerVolumeSpecName "kube-api-access-v745x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:36.759509 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.759461    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v745x\" (UniqueName: \"kubernetes.io/projected/87a10b27-221f-4153-ae74-132fdf500a63-kube-api-access-v745x\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:23:36.759509 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.759502    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a10b27-221f-4153-ae74-132fdf500a63-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Sep 18 20:23:36.794173 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.794141    2580 generic.go:356] "Generic (PLEG): container finished" podID="87a10b27-221f-4153-ae74-132fdf500a63" containerID="0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c" exitCode=0
Sep 18 20:23:36.794361 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.794194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"
Sep 18 20:23:36.794361 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.794217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" event={"ID":"87a10b27-221f-4153-ae74-132fdf500a63","Type":"ContainerDied","Data":"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"}
Sep 18 20:23:36.794361 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.794267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f" event={"ID":"87a10b27-221f-4153-ae74-132fdf500a63","Type":"ContainerDied","Data":"7b26cff63cfa2fffc2abe7edbb55b03600c17e6d23fbbb90d54d0b7448493cd3"}
Sep 18 20:23:36.794361 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.794290    2580 scope.go:117] "RemoveContainer" containerID="0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"
Sep 18 20:23:36.803114 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.803098    2580 scope.go:117] "RemoveContainer" containerID="0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"
Sep 18 20:23:36.803389 ip-10-0-138-2 kubenswrapper[2580]: E0918 20:23:36.803372    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c\": container with ID starting with 0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c not found: ID does not exist" containerID="0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"
Sep 18 20:23:36.803431 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.803396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c"} err="failed to get container status \"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c\": rpc error: code = NotFound desc = could not find container \"0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c\": container with ID starting with 0d64e089b0afece2cca5bbc0cc13269e5aaaa22ce19348d81999fc736f74878c not found: ID does not exist"
Sep 18 20:23:36.817932 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.817810    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:23:36.819757 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:36.819738    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgd76f"]
Sep 18 20:23:37.294759 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:37.294729    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a10b27-221f-4153-ae74-132fdf500a63" path="/var/lib/kubelet/pods/87a10b27-221f-4153-ae74-132fdf500a63/volumes"
Sep 18 20:23:41.267505 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:41.267470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:48.009022 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:48.008970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:54.632981 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:54.632944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:54.722896 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:23:54.722863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:01.573107 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:01.573073    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:24:01.587024 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:01.587004    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2pp8z_0e9e5c67-f3b4-4baa-b5a5-ce16acc418d5/ovn-acl-logging/0.log"
Sep 18 20:24:19.426847 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:19.426773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:20.442910 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:20.442884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:24.336799 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:24.336764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:24.561678 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:24.561615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:27.014229 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:27.014194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:29.359633 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:29.359573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:32.930949 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:32.930919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:33.008462 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:24:33.008433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:06.635083 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:06.635037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:06.779773 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:06.779737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:07.050800 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:07.050766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:07.164943 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:07.164913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:55.785411 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:55.785364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:57.359275 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:57.359233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:57.481227 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:25:57.481190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:03.132352 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:03.132317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.352120 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:09.352083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:09.436756 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:09.436718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:22.474142 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:22.474103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bgpc8_fdd752a6-29ef-42f8-9930-9d1fcdaa389e/discovery/0.log"
Sep 18 20:26:22.589122 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:22.589089    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7887dff69b-grctd_d2d172c8-0c5e-488e-bf1b-00946e7f692d/router/0.log"
Sep 18 20:26:23.498037 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:23.498003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bgpc8_fdd752a6-29ef-42f8-9930-9d1fcdaa389e/discovery/0.log"
Sep 18 20:26:23.566716 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:23.566689    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7887dff69b-grctd_d2d172c8-0c5e-488e-bf1b-00946e7f692d/router/0.log"
Sep 18 20:26:24.467566 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:24.467538    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wn4r4_cee93929-03a4-4077-96ad-0809011aa6f3/manager/0.log"
Sep 18 20:26:24.659918 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:24.659888    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8ndkz_395caf8f-4cd7-4c55-bed4-3219dcf1ee3c/manager/0.log"
Sep 18 20:26:26.773293 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:26.773260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:26.985069 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:26.985038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:27.088896 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:27.088864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:30.019116 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:30.019084    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lx4lc_b3313fdc-9479-4f30-8880-121063d20652/global-pull-secret-syncer/0.log"
Sep 18 20:26:30.130481 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:30.130448    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log"
Sep 18 20:26:30.279063 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:30.278981    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-2.ec2.internal_8cbeccd17f22a204da485376383e5ad9/haproxy/0.log"
Sep 18 20:26:33.669268 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:33.669236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:33.832833 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:33.832801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:34.819057 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:34.819017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sxjj_addcf4d5-8a05-4680-a602-5af3861d16c7/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:35.136455 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:35.136378    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wn4r4_cee93929-03a4-4077-96ad-0809011aa6f3/manager/0.log"
Sep 18 20:26:35.500057 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:35.499969    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8ndkz_395caf8f-4cd7-4c55-bed4-3219dcf1ee3c/manager/0.log"
Sep 18 20:26:37.518436 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:37.518402    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-msmt9_e871430e-31d9-4f13-8462-e45a0b76452d/cluster-monitoring-operator/0.log"
Sep 18 20:26:37.571479 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:37.571451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-state-metrics/0.log"
Sep 18 20:26:37.599448 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:37.599423    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-rbac-proxy-main/0.log"
Sep 18 20:26:37.634153 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:37.634129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-kzgfg_bce68abf-eb0c-4309-b087-34a2b3a391a5/kube-rbac-proxy-self/0.log"
Sep 18 20:26:37.739874 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:37.739830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-kp5bh_b002d540-be83-429f-8b5d-7e5698f302d9/monitoring-plugin/0.log"
Sep 18 20:26:38.008311 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.008283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/node-exporter/0.log"
Sep 18 20:26:38.040512 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.040487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/kube-rbac-proxy/0.log"
Sep 18 20:26:38.070815 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.070788    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wll88_f8414775-82dc-4249-b301-5ae01f963546/init-textfile/0.log"
Sep 18 20:26:38.119648 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.119623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/kube-rbac-proxy-main/0.log"
Sep 18 20:26:38.152624 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.152577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/kube-rbac-proxy-self/0.log"
Sep 18 20:26:38.193765 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.193732    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q28qv_b2f23b3d-db6f-411c-9f4c-604218090fec/openshift-state-metrics/0.log"
Sep 18 20:26:38.714462 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.714434    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-qx665_40e45865-d562-4582-b91d-0eb394419b1c/prometheus-operator-admission-webhook/0.log"
Sep 18 20:26:38.757818 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.757789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94bc99d-7kzjv_0534aae7-245b-4984-8f24-1a8ce2fa89a4/telemeter-client/0.log"
Sep 18 20:26:38.795483 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.795450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94bc99d-7kzjv_0534aae7-245b-4984-8f24-1a8ce2fa89a4/reload/0.log"
Sep 18 20:26:38.839406 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.839384    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94bc99d-7kzjv_0534aae7-245b-4984-8f24-1a8ce2fa89a4/kube-rbac-proxy/0.log"
Sep 18 20:26:38.881968 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.881938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/thanos-query/0.log"
Sep 18 20:26:38.916653 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.916622    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-web/0.log"
Sep 18 20:26:38.950746 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.950719    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy/0.log"
Sep 18 20:26:38.983157 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:38.983086    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/prom-label-proxy/0.log"
Sep 18 20:26:39.012382 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.012324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-rules/0.log"
Sep 18 20:26:39.040980 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.040953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d47b47bb-8skct_84038c7f-0898-42f5-89ef-6025e7f2aa19/kube-rbac-proxy-metrics/0.log"
Sep 18 20:26:39.319716 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.319686    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"]
Sep 18 20:26:39.320065 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.320052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main"
Sep 18 20:26:39.320122 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.320066    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main"
Sep 18 20:26:39.320157 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.320130    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a10b27-221f-4153-ae74-132fdf500a63" containerName="main"
Sep 18 20:26:39.323276 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.323257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.326390 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.326369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"openshift-service-ca.crt\""
Sep 18 20:26:39.326498 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.326370    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sggjm\"/\"default-dockercfg-gqtsx\""
Sep 18 20:26:39.328326 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.328310    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"kube-root-ca.crt\""
Sep 18 20:26:39.333851 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.333823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"]
Sep 18 20:26:39.413840 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.413810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxwm\" (UniqueName: \"kubernetes.io/projected/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-kube-api-access-8bxwm\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.414011 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.413849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-podres\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.414011 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.413880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-proc\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.414011 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.413931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-sys\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.414011 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.413968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-lib-modules\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515028 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.514994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-lib-modules\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515188 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxwm\" (UniqueName: \"kubernetes.io/projected/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-kube-api-access-8bxwm\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515188 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-podres\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515188 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-proc\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515188 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-sys\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515327 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-lib-modules\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515327 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-proc\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515327 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-sys\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.515327 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.515234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-podres\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.525772 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.525752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxwm\" (UniqueName: \"kubernetes.io/projected/0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb-kube-api-access-8bxwm\") pod \"perf-node-gather-daemonset-r6m5w\" (UID: \"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.633732 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.633663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:39.913922 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:39.913898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"]
Sep 18 20:26:39.915811 ip-10-0-138-2 kubenswrapper[2580]: W0918 20:26:39.915780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c48c1b1_b0c4_45cd_9c99_9b6e1ea1fedb.slice/crio-ed6bdca28a7c868ac193823b77f7f6e8b7545170c075c5338bec8a306faf050e WatchSource:0}: Error finding container ed6bdca28a7c868ac193823b77f7f6e8b7545170c075c5338bec8a306faf050e: Status 404 returned error can't find the container with id ed6bdca28a7c868ac193823b77f7f6e8b7545170c075c5338bec8a306faf050e
Sep 18 20:26:40.676465 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:40.676430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w" event={"ID":"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb","Type":"ContainerStarted","Data":"381e5577bfeb3949e6447705e62980977b28f202637360fb14062068a390884a"}
Sep 18 20:26:40.676669 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:40.676480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w" event={"ID":"0c48c1b1-b0c4-45cd-9c99-9b6e1ea1fedb","Type":"ContainerStarted","Data":"ed6bdca28a7c868ac193823b77f7f6e8b7545170c075c5338bec8a306faf050e"}
Sep 18 20:26:40.676669 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:40.676532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:40.700197 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:40.700152    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w" podStartSLOduration=1.70013836 podStartE2EDuration="1.70013836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:40.699183785 +0000 UTC m=+4361.486464190" watchObservedRunningTime="2026-09-18 20:26:40.70013836 +0000 UTC m=+4361.487418762"
Sep 18 20:26:41.303152 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:41.303126    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f86c8bf58-8b62j_ebe9c100-5cd9-4beb-9655-2c30dcc09401/console/0.log"
Sep 18 20:26:41.354459 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:41.354435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-q5jdx_8030ffbd-33ee-45c7-858b-9491d5902930/download-server/0.log"
Sep 18 20:26:43.230823 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:43.230798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7nph_619098d1-6642-4cb5-814d-236b8dfeee2f/dns/0.log"
Sep 18 20:26:43.261251 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:43.261226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7nph_619098d1-6642-4cb5-814d-236b8dfeee2f/kube-rbac-proxy/0.log"
Sep 18 20:26:43.437774 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:43.437745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xk6nb_0abe42a6-25e0-4573-84ca-2f16520a9382/dns-node-resolver/0.log"
Sep 18 20:26:44.080689 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:44.080657    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58cccc5b88-8kqgv_1160973b-4bf3-4d10-90c5-c733bfadb18f/registry/0.log"
Sep 18 20:26:44.194675 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:44.194637    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6lcg_458496de-2e73-49dc-92fa-743221adf3b4/node-ca/0.log"
Sep 18 20:26:45.478232 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:45.478192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bgpc8_fdd752a6-29ef-42f8-9930-9d1fcdaa389e/discovery/0.log"
Sep 18 20:26:45.606083 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:45.606034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7887dff69b-grctd_d2d172c8-0c5e-488e-bf1b-00946e7f692d/router/0.log"
Sep 18 20:26:46.250475 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:46.250444    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l926h_e58327ff-f218-4763-8ddc-9367f2b79ee8/serve-healthcheck-canary/0.log"
Sep 18 20:26:46.691039 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:46.691006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-r6m5w"
Sep 18 20:26:47.175553 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:47.175522    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndhng_f031cd97-0d94-4fed-90c6-69befc4a61e8/kube-rbac-proxy/0.log"
Sep 18 20:26:47.227039 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:47.227017    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndhng_f031cd97-0d94-4fed-90c6-69befc4a61e8/exporter/0.log"
Sep 18 20:26:47.302216 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:47.302193    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndhng_f031cd97-0d94-4fed-90c6-69befc4a61e8/extractor/0.log"
Sep 18 20:26:49.548095 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:49.548071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bgpc8_fdd752a6-29ef-42f8-9930-9d1fcdaa389e/discovery/0.log"
Sep 18 20:26:49.748541 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:49.748472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7887dff69b-grctd_d2d172c8-0c5e-488e-bf1b-00946e7f692d/router/0.log"
Sep 18 20:26:50.931679 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:50.931652    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bgpc8_fdd752a6-29ef-42f8-9930-9d1fcdaa389e/discovery/0.log"
Sep 18 20:26:51.044362 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:51.044337    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7887dff69b-grctd_d2d172c8-0c5e-488e-bf1b-00946e7f692d/router/0.log"
Sep 18 20:26:52.053251 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:52.053223    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5ccfd4fdb4-p2p95_00f7e83b-ee05-4475-9218-3be06c564f7e/manager/0.log"
Sep 18 20:26:52.589908 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:52.589880    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-65474cf8d5-6msh7_ee309e4c-2eb0-4bdd-9483-ec5a7fbd22e9/manager/0.log"
Sep 18 20:26:52.709598 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:52.709551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-29w5w_a7972b76-3801-45cd-89fc-8a932555f309/server/0.log"
Sep 18 20:26:53.582764 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:53.582728    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-24z7c_41294152-b30a-46f7-b1a0-8b13ede112b6/manager/0.log"
Sep 18 20:26:53.799855 ip-10-0-138-2 kubenswrapper[2580]: I0918 20:26:53.799819    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qqcnr_2270b2cf-aaee-4ef4-b0e4-4bc15e576c33/seaweedfs/0.log"